Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/alpstable/gidari

Transport web data to local/remote storage using Gidari

https://github.com/alpstable/gidari

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.4%) to scientific vocabulary

Keywords

api-wrapper csv data-science go http mongodb storage

Keywords from Contributors

interactive projection sequences data-engineering autograding hacking shellcodes archival modular scan
Last synced: 6 months ago · JSON representation

Repository

Transport web data to local/remote storage using Gidari

Basic Info
  • Host: GitHub
  • Owner: alpstable
  • License: apache-2.0
  • Language: Go
  • Default Branch: main
  • Homepage:
  • Size: 31.8 MB
Statistics
  • Stars: 29
  • Watchers: 2
  • Forks: 29
  • Open Issues: 8
  • Releases: 4
Topics
api-wrapper csv data-science go http mongodb storage
Created over 3 years ago · Last pushed about 2 years ago
Metadata Files
Readme Contributing License

README.md

Gidari

PkgGoDev Build Status Go Report Card Discord

Gidari is a library for batch querying data and persisting the results to local storage.

Installation

sh go get github.com/alpstable/gidari@latest

For information on using the CLI, see here.

Usage

Gidari supports HTTP and network socket services. There are two ways to use an HTTP service:

  1. Iterate over http.Response data, for pre-defined http.Requests.
  2. Define a writer to concurrently "write" response data for pre-defined http.Requests.

See the Go Docs or Web-to-Storage Examples section for examples.

Network sockets involves subscribing to a socket (such a web socket) and continuously iterating over the data via ReadWriter interface. The results would then be sent to a user-defined ListWriter to be stored. See the Go Docs or Network Socket Examples section for examples.

Authenticating HTTP Requests

| Protocol | Parameters | Description | |--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Basic | Requires a username/email and password | The server will authorize the request only if it can validate the user-ID and password for the protection space of the Request-URI | | Coinbase | Requires a key, passphrase and secrete generated in the Coinbase Exchange GUI | The Coinbase API requires a CB-ACCESS-SIGN header that is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string from a timestamp + method + requestPath + body combination | | Kraken | Requires key and secret generated in the Kraken Pro GUI | The Kraken spot API uses a custom authentication algorithm that is based on a combination of API key, nonce, and message signature. The signature is generated using a hash-based message authentication code (HMAC) with SHA-512 as the hash function. |

Web-to-Storage Examples

| Data Type | Writer | Example | Description | |-----------|-------------------------------------------------|----------------------------------------------|-------------------------------------------------------------------| | CSV | csvpb | examples/csvp | Use the HTTPService to write web API data to stdout as CSV | | MongoDB Document | mongopb | exmaples/mongopb | Use the HTTPService to write web API data to a MongoDB Collection |

Network Socket Examples

| Type | Writer | Example | Description | |------|---------------|------------------------------------------|-----------------------------------------------------------| | WebSocket |fmt.Println | examples/websocket | Subscribe to the Coinbase WebSocket and print the results |

Contributing

Follow this guide for information on contributing.

Resources

Owner

  • Name: Alpstable
  • Login: alpstable
  • Kind: organization

GitHub Events

Total
Last Year

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 717
  • Total Committers: 37
  • Avg Commits per committer: 19.378
  • Development Distribution Score (DDS): 0.287
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Preston Vasquez 2****z 511
Alexander Tsapkov a****v@g****m 30
Conor Fleming c****3@g****m 26
Preston Vasquez p****z@i****m 21
Joseph Peloquin 4****9 18
Rakshat Kashyap k****t@g****m 15
Gaurav Gahlot g****7@g****m 14
Amogh a****i@c****m 11
Preston Vasquez p****0@g****m 7
Pankhudi Bhonsle p****8@g****m 7
Fahim Bagar f****r@g****m 6
McCune1224 a****4@g****m 6
ashu adhana a****a@g****m 6
Kumar Sonu 9****t 5
Cédric L'homme p****c@l****m 4
Amogh Desai a****9@g****m 3
SoloDv f****9@g****m 3
Syerikjan Kh k****n@g****m 2
Tsvetoslav Dimov b****i@g****m 2
Vinício Augusto v****n@g****m 2
Joseph Peloquin c****9@g****m 2
Bruno Ribeiro da Silva b****d@g****m 1
Henrik Nilsenius H****k@n****u 1
Janet Thieu j****u@g****m 1
dependabot[bot] 4****] 1
Conor Fleming 3****g 1
Nikhil Vaidyar n****r@b****m 1
Patrik Hjortshøj Lindberg p****g@g****m 1
Peterson Domingos m****n@g****m 1
PiotrLewandowski323 l****3@g****m 1
and 7 more...
Committer Domains (Top 20 + Academic)
proton.me: 1 impactnorth.com: 1 biofourmis.com: 1 nilsenius.nu: 1 l-homme.com: 1 cloudera.com: 1

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 62
  • Total pull requests: 41
  • Average time to close issues: 2 months
  • Average time to close pull requests: 7 days
  • Total issue authors: 1
  • Total pull request authors: 7
  • Average comments per issue: 1.87
  • Average comments per pull request: 0.56
  • Merged pull requests: 36
  • Bot issues: 0
  • Bot pull requests: 3
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 1
View more stats
Top Authors
Issue Authors
  • prestonvasquez (62)
Pull Request Authors
  • prestonvasquez (28)
  • Conor-Fleming (5)
  • fahimbagar (2)
  • dependabot[bot] (2)
  • NotAtitYet (1)
  • ash-kamrip (1)
  • rish-singhal (1)
Top Labels
Issue Labels
enhancement (26) good first issue (18) documentation (17) hacktoberfest (14) 0.2.0 (14) 0.4.0 (8) tests (6) bug (6) 0.1.1 (5) architecture (5) 0.3.0 (4) research (3) question (3) breaking (3) ci (3) other (2) non-go (2) 0.2.1 (2) performance (2) feature (2) investigation (1) 0.1.2 (1)
Pull Request Labels
hacktoberfest-accepted (2) dependencies (2) codeball:approved (1)

Packages

  • Total packages: 5
  • Total downloads: unknown
  • Total dependent packages: 5
    (may contain duplicates)
  • Total dependent repositories: 2
    (may contain duplicates)
  • Total versions: 6
proxy.golang.org: github.com/alpstable/gidari

Package gidari provides a "web-to-storage" API for batch querying web APIs and persisting the resulting data.

  • Versions: 5
  • Dependent Packages: 5
  • Dependent Repositories: 2
Rankings
Dependent packages count: 2.5%
Dependent repos count: 3.5%
Forks count: 4.0%
Average: 4.3%
Stargazers count: 7.1%
Last synced: 6 months ago
proxy.golang.org: github.com/alpstable/gidari/cmd/gidari

Copyright 2022 The Gidari Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Forks count: 2.3%
Stargazers count: 4.0%
Average: 5.7%
Dependent packages count: 7.0%
Dependent repos count: 9.3%
Last synced: 8 months ago
proxy.golang.org: github.com/alpstable/gidari/examples/csvpb
  • Versions: 0
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 8.8%
Average: 9.4%
Dependent repos count: 9.9%
Last synced: 6 months ago
proxy.golang.org: github.com/alpstable/gidari/examples/websocket
  • Versions: 0
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 8.9%
Average: 9.4%
Dependent repos count: 10.0%
Last synced: 8 months ago
proxy.golang.org: github.com/alpstable/gidari/examples/mongopb
  • Versions: 0
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 8.9%
Average: 9.4%
Dependent repos count: 10.0%
Last synced: 6 months ago

Dependencies

go.mod go
  • github.com/golang/snappy v0.0.1
  • github.com/google/uuid v1.1.2
  • github.com/inconshreveable/mousetrap v1.0.0
  • github.com/klauspost/compress v1.13.6
  • github.com/kr/text v0.2.0
  • github.com/lib/pq v1.10.6
  • github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe
  • github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
  • github.com/pkg/errors v0.9.1
  • github.com/sirupsen/logrus v1.9.0
  • github.com/spf13/cobra v1.5.0
  • github.com/spf13/pflag v1.0.5
  • github.com/stretchr/testify v1.8.0
  • github.com/xdg-go/pbkdf2 v1.0.0
  • github.com/xdg-go/scram v1.1.1
  • github.com/xdg-go/stringprep v1.0.3
  • github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d
  • go.mongodb.org/mongo-driver v1.10.0
  • golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
  • golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
  • golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8
  • golang.org/x/text v0.3.7
  • golang.org/x/time v0.0.0-20220722155302-e5dcc9cfc0b9
  • golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
  • google.golang.org/protobuf v1.28.0
  • gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
  • gopkg.in/yaml.v2 v2.4.0
go.sum go
  • github.com/cpuguy83/go-md2man/v2 v2.0.2
  • github.com/creack/pty v1.1.9
  • github.com/davecgh/go-spew v1.1.0
  • github.com/davecgh/go-spew v1.1.1
  • github.com/golang/protobuf v1.5.0
  • github.com/golang/snappy v0.0.1
  • github.com/google/go-cmp v0.5.2
  • github.com/google/go-cmp v0.5.5
  • github.com/google/uuid v1.1.2
  • github.com/inconshreveable/mousetrap v1.0.0
  • github.com/klauspost/compress v1.13.6
  • github.com/kr/pretty v0.1.0
  • github.com/kr/pty v1.1.1
  • github.com/kr/text v0.1.0
  • github.com/kr/text v0.2.0
  • github.com/lib/pq v1.10.6
  • github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe
  • github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
  • github.com/pkg/errors v0.9.1
  • github.com/pmezard/go-difflib v1.0.0
  • github.com/russross/blackfriday/v2 v2.1.0
  • github.com/sirupsen/logrus v1.9.0
  • github.com/spf13/cobra v1.5.0
  • github.com/spf13/pflag v1.0.5
  • github.com/stretchr/objx v0.1.0
  • github.com/stretchr/objx v0.4.0
  • github.com/stretchr/testify v1.6.1
  • github.com/stretchr/testify v1.7.0
  • github.com/stretchr/testify v1.7.1
  • github.com/stretchr/testify v1.8.0
  • github.com/tidwall/pretty v1.0.0
  • github.com/xdg-go/pbkdf2 v1.0.0
  • github.com/xdg-go/scram v1.1.1
  • github.com/xdg-go/stringprep v1.0.3
  • github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d
  • go.mongodb.org/mongo-driver v1.10.0
  • golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
  • golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2
  • golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
  • golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
  • golang.org/x/sys v0.0.0-20201119102817-f84b799fce68
  • golang.org/x/sys v0.0.0-20210423082822-04245dca01da
  • golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1
  • golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8
  • golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
  • golang.org/x/text v0.3.6
  • golang.org/x/text v0.3.7
  • golang.org/x/time v0.0.0-20220722155302-e5dcc9cfc0b9
  • golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e
  • golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
  • golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
  • google.golang.org/protobuf v1.26.0-rc.1
  • google.golang.org/protobuf v1.28.0
  • gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
  • gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
  • gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
  • gopkg.in/yaml.v2 v2.4.0
  • gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
  • gopkg.in/yaml.v3 v3.0.1
.github/workflows/ci.yml actions
  • actions/checkout v3 composite
  • actions/setup-go v3 composite
.github/workflows/codeball.yml actions
  • sturdy-dev/codeball-action v2 composite
.github/workflows/codeql.yml actions
  • actions/checkout v3 composite
  • github/codeql-action/analyze v2 composite
  • github/codeql-action/autobuild v2 composite
  • github/codeql-action/init v2 composite
.github/workflows/dependency-review.yml actions
  • actions/checkout v3 composite
  • actions/dependency-review-action v2 composite
.github/workflows/golangci-lint.yml actions
  • actions/checkout v3 composite
  • actions/setup-go v3 composite
  • golangci/golangci-lint-action v3 composite
examples/csvpb/go.mod go
  • github.com/alpstable/csvpb v0.1.1
  • github.com/alpstable/gidari v0.1.0
  • golang.org/x/time v0.3.0
  • google.golang.org/protobuf v1.28.1
examples/csvpb/go.sum go
  • github.com/alpstable/csvpb v0.1.0
  • github.com/alpstable/csvpb v0.1.1
  • github.com/alpstable/gidari v0.1.0
  • github.com/golang/protobuf v1.5.0
  • github.com/google/go-cmp v0.5.5
  • golang.org/x/time v0.3.0
  • golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
  • golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
  • google.golang.org/protobuf v1.26.0-rc.1
  • google.golang.org/protobuf v1.28.1
examples/mongopb/go.mod go
  • github.com/alpstable/gidari v0.1.1
  • github.com/alpstable/mongopb v0.1.1
  • github.com/golang/snappy v0.0.1
  • github.com/klauspost/compress v1.13.6
  • github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe
  • github.com/pkg/errors v0.9.1
  • github.com/xdg-go/pbkdf2 v1.0.0
  • github.com/xdg-go/scram v1.1.1
  • github.com/xdg-go/stringprep v1.0.3
  • github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d
  • go.mongodb.org/mongo-driver v1.11.2
  • golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
  • golang.org/x/sync v0.1.0
  • golang.org/x/text v0.3.8
  • golang.org/x/time v0.3.0
  • google.golang.org/protobuf v1.28.1
examples/mongopb/go.sum go
  • github.com/alpstable/mongopb v0.1.1
  • github.com/davecgh/go-spew v1.1.0
  • github.com/davecgh/go-spew v1.1.1
  • github.com/golang/protobuf v1.5.0
  • github.com/golang/snappy v0.0.1
  • github.com/google/go-cmp v0.5.2
  • github.com/google/go-cmp v0.5.5
  • github.com/klauspost/compress v1.13.6
  • github.com/kr/pretty v0.1.0
  • github.com/kr/pty v1.1.1
  • github.com/kr/text v0.1.0
  • github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe
  • github.com/pkg/errors v0.9.1
  • github.com/pmezard/go-difflib v1.0.0
  • github.com/stretchr/objx v0.1.0
  • github.com/stretchr/testify v1.6.1
  • github.com/tidwall/pretty v1.0.0
  • github.com/xdg-go/pbkdf2 v1.0.0
  • github.com/xdg-go/scram v1.1.1
  • github.com/xdg-go/stringprep v1.0.3
  • github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d
  • go.mongodb.org/mongo-driver v1.11.2
  • golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
  • golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2
  • golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
  • golang.org/x/sync v0.1.0
  • golang.org/x/sys v0.0.0-20201119102817-f84b799fce68
  • golang.org/x/sys v0.0.0-20210423082822-04245dca01da
  • golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1
  • golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
  • golang.org/x/text v0.3.6
  • golang.org/x/text v0.3.7
  • golang.org/x/text v0.3.8
  • golang.org/x/time v0.3.0
  • golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e
  • golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
  • golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
  • google.golang.org/protobuf v1.26.0-rc.1
  • google.golang.org/protobuf v1.28.1
  • gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
  • gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
  • gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
  • gopkg.in/yaml.v3 v3.0.1
examples/websocket/go.mod go
  • github.com/alpstable/gidari v0.0.0-00010101000000-000000000000
  • golang.org/x/net v0.9.0
  • golang.org/x/time v0.3.0
  • google.golang.org/protobuf v1.30.0
examples/websocket/go.sum go
  • github.com/golang/protobuf v1.5.0
  • github.com/google/go-cmp v0.5.5
  • golang.org/x/net v0.9.0
  • golang.org/x/time v0.3.0
  • golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
  • golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
  • google.golang.org/protobuf v1.26.0-rc.1
  • google.golang.org/protobuf v1.30.0