dataretrieval-python
Python package for retrieving water data from USGS or the multi-agency Water Quality Portal
Science Score: 57.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 2 DOI reference(s) in README -
○Academic publication links
-
✓Committers with academic emails
7 of 20 committers (35.0%) from academic institutions -
✓Institutional organization owner
Organization doi-usgs has institutional domain (www.usgs.gov) -
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.3%) to scientific vocabulary
Keywords from Contributors
Repository
Python package for retrieving water data from USGS or the multi-agency Water Quality Portal
Basic Info
- Host: GitHub
- Owner: DOI-USGS
- License: other
- Language: Python
- Default Branch: main
- Homepage: https://doi-usgs.github.io/dataretrieval-python/
- Size: 15.8 MB
Statistics
- Stars: 201
- Watchers: 11
- Forks: 50
- Open Issues: 30
- Releases: 16
Metadata Files
README.md
dataretrieval: Download hydrologic data
:warning: USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting legacy=False in the functions in the wqp module.
To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html
:mega: 09/03/2024: The groundwater levels service has switched endpoints, and dataretrieval was updated accordingly in v1.0.10. Older versions using the discontinued endpoint will return 503 errors for nwis.get_gwlevels or the service='gwlevels' argument. Visit Water Data For the Nation for more information.
What is dataretrieval?
dataretrieval was created to simplify the process of loading hydrologic data into the Python environment.
Like the original R version dataRetrieval,
it is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology
data that are available on the Web, as well as data from the Water
Quality Portal (WQP), which currently houses water quality data from the
Environmental Protection Agency (EPA), U.S. Department of Agriculture
(USDA), and USGS. Direct USGS data is obtained from a service called the
National Water Information System (NWIS).
Note that the python version is not a direct port of the original: it attempts to reproduce the functionality of the R package, though its organization and interface often differ.
If there's a hydrologic or environmental data portal that you'd like dataretrieval to work with, raise it as an issue.
Here's an example using dataretrieval to retrieve data from the National Water Information System (NWIS).
```python
first import the functions for downloading data from NWIS
import dataretrieval.nwis as nwis
specify the USGS site code for which we want data.
site = '03339000'
get instantaneous values (iv)
df = nwis.get_record(sites=site, service='iv', start='2017-12-31', end='2018-01-01')
get basic info about the site
df2 = nwis.get_record(sites=site, service='site') ``` Services available from NWIS include: - instantaneous values (iv) - daily values (dv) - statistics (stat) - site info (site) - discharge peaks (peaks) - discharge measurements (measurements)
Water quality data are available from: - Samples - Discrete USGS water quality data only - Water Quality Portal - Discrete water quality data from USGS and EPA. Older data are available in the legacy WQX version 2 format; all data are available in the beta WQX3.0 format.
To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example, this function call:
python
nwis.get_record(sites='03339000', service='dv', start='2017-12-31', parameterCd='00060')
...will download daily data with the parameter code 00060 (discharge).
Accessing the "Internal" NWIS
If you're connected to the USGS network, dataretrieval call pull from the internal (non-public) NWIS interface.
Most dataretrieval functions pass kwargs directly to NWIS's REST API, which provides simple access to internal data; simply specify "access='3'".
For example
python
nwis.get_record(sites='05404147',service='iv', start='2021-01-01', end='2021-3-01', access='3')
More services and documentation to come!
Quick start
dataretrieval can be installed using pip:
$ python3 -m pip install -U dataretrieval
or conda:
$ conda install -c conda-forge dataretrieval
More examples of use are include in demos.
Issue tracker
Please report any bugs and enhancement ideas using the dataretrieval issue tracker:
https://github.com/USGS-python/dataretrieval/issues
Feel free to also ask questions on the tracker.
Contributing
Any help in testing, development, documentation and other tasks is welcome. For more details, see the file CONTRIBUTING.md.
Need help?
The Water Mission Area of the USGS supports the development and maintenance of dataretrieval. Any questions can be directed to the Computational Tools team at
comptools@usgs.gov.
Resources are available primarily for maintenance and responding to user questions.
Priorities on the development of new features are determined by the dataretrieval development team.
Acknowledgments
This material is partially based upon work supported by the National Science Foundation (NSF) under award 1931297. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
Disclaimer
This software is preliminary or provisional and is subject to revision. It is being provided to meet the need for timely best science. The software has not received final approval by the U.S. Geological Survey (USGS). No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty. The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.
Citation
Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023, dataretrieval (Python): a Python package for discovering and retrieving water data available from U.S. federal hydrologic web services: U.S. Geological Survey software release, https://doi.org/10.5066/P94I5TX3.
Owner
- Name: U.S. Geological Survey
- Login: DOI-USGS
- Kind: organization
- Email: gs_help_git@usgs.gov
- Location: United States of America
- Website: https://www.usgs.gov/
- Twitter: USGS
- Repositories: 59
- Profile: https://github.com/DOI-USGS
By integrating our diverse scientific expertise, we understand complex natural science phenomena and provide scientific products that lead to solutions.
GitHub Events
Total
- Create event: 3
- Issues event: 7
- Release event: 2
- Watch event: 37
- Issue comment event: 31
- Push event: 9
- Pull request review comment event: 5
- Pull request review event: 11
- Pull request event: 10
- Fork event: 10
Last Year
- Create event: 3
- Issues event: 7
- Release event: 2
- Watch event: 37
- Issue comment event: 31
- Push event: 9
- Pull request review comment event: 5
- Pull request review event: 11
- Pull request event: 10
- Fork event: 10
Committers
Last synced: 6 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Timothy Hodson | 3****s | 73 |
| Elise Hinman | e****n@u****v | 40 |
| J. Hariharan | j****n@u****v | 22 |
| Scott Black | s****k@u****u | 8 |
| Pabitra Dash | p****a@h****m | 5 |
| Joseph Stachelek | j****a | 4 |
| Ibrahim El Merehbi | e****i | 2 |
| Doug Dennis | d****r@g****m | 2 |
| Joshua Larsen | j****n@u****v | 2 |
| Austin Raney | a****y@n****v | 1 |
| Camilo J. Bastidas Pacheco | c****s@g****m | 1 |
| Dave Tapley | d****e@t****m | 1 |
| David Blodgett | d****t@u****v | 1 |
| Edwin | 3****c | 1 |
| Joe Zemmels (he/him) | j****s@g****m | 1 |
| Justin Bousquin | j****n@y****m | 1 |
| Keith Doore | 5****e | 1 |
| Lee Stanish | 8****s | 1 |
| MLR | m****1@g****m | 1 |
| mnfienen | m****n@u****v | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 97
- Total pull requests: 84
- Average time to close issues: 4 months
- Average time to close pull requests: 13 days
- Total issue authors: 49
- Total pull request authors: 20
- Average comments per issue: 2.14
- Average comments per pull request: 1.81
- Merged pull requests: 74
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 13
- Pull requests: 14
- Average time to close issues: about 1 month
- Average time to close pull requests: 11 days
- Issue authors: 7
- Pull request authors: 7
- Average comments per issue: 1.31
- Average comments per pull request: 3.07
- Merged pull requests: 12
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- thodson-usgs (15)
- lstanish-usgs (7)
- ehinman (7)
- SorooshMani-NOAA (5)
- davetapley (5)
- elbeejay (5)
- putmanannie (4)
- pkdash (2)
- jpouliot10 (2)
- msweier (2)
- jsta (2)
- jreniel (1)
- oconnelljon (1)
- hwreeves-USGS (1)
- horsburgh (1)
Pull Request Authors
- thodson-usgs (26)
- elbeejay (21)
- ehinman (20)
- pkdash (6)
- jsta (5)
- sblack-usu (4)
- jlarsen-usgs (4)
- davetapley (3)
- elmerehbi (3)
- cjbas22 (3)
- dblodgett-usgs (2)
- aaraney (2)
- kjdoore (2)
- lstanish-usgs (2)
- edsaac (2)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 3
-
Total downloads:
- pypi 8,353 last-month
-
Total dependent packages: 8
(may contain duplicates) -
Total dependent repositories: 11
(may contain duplicates) - Total versions: 45
- Total maintainers: 2
pypi.org: dataretrieval
Discover and retrieve water data from U.S. federal hydrologic web services.
- Documentation: https://dataretrieval.readthedocs.io/
- License: License ======= Unless otherwise noted, this project is in the public domain in the United States because it contains materials that originally came from the United States Geological Survey, an agency of the United States Department of Interior. For more information, see the official USGS copyright policy at https://www.usgs.gov/information-policies-and-instructions/copyrights-and-credits Additionally, we waive copyright and related rights in the work worldwide through the CC0 1.0 Universal public domain dedication. CC0 1.0 Universal Summary ------------------------- This is a human-readable summary of the [Legal Code (read the full text)][1]. ### No Copyright The person who associated a work with this deed has dedicated the work to the public domain by waiving all of his or her rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law. You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission. ### Other Information In no way are the patent or trademark rights of any person affected by CC0, nor are the rights that other persons may have in the work or in how the work is used, such as publicity or privacy rights. Unless expressly stated otherwise, the person who associated a work with this deed makes no warranties about the work, and disclaims liability for all uses of the work, to the fullest extent permitted by applicable law. When using or citing the work, you should not imply endorsement by the author or the affirmer. [1]: https://creativecommons.org/publicdomain/zero/1.0/legalcode
-
Latest release: 1.0.12
published 10 months ago
Rankings
Maintainers (2)
proxy.golang.org: github.com/DOI-USGS/dataretrieval-python
- Documentation: https://pkg.go.dev/github.com/DOI-USGS/dataretrieval-python#section-documentation
- License: other
-
Latest release: v1.0.12
published 10 months ago
Rankings
proxy.golang.org: github.com/doi-usgs/dataretrieval-python
- Documentation: https://pkg.go.dev/github.com/doi-usgs/dataretrieval-python#section-documentation
- License: other
-
Latest release: v1.0.12
published 10 months ago
Rankings
Dependencies
- actions/checkout v2 composite
- actions/setup-python v2 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- pypa/gh-action-pypi-publish 27b31702a0e7fc50959f5ad993c78deac1bdfc29 composite
- JamesIves/github-pages-deploy-action v4.4.1 composite
- actions/checkout v3 composite
- coverage * development
- flake8 * development
- ipykernel * development
- ipython * development
- matplotlib * development
- nbsphinx * development
- nbsphinx_link * development
- numpy * development
- pandas * development
- pytest * development
- python-dateutil * development
- requests * development
- requests-mock * development
- scipy * development
- sphinx * development
- sphinx-rtd-theme * development
- numpy *
- pandas *
- python-dateutil *
- requests *