Data

Tools

Indexes

Applications

Experiments

Open Source Science

hydrogauge

https://github.com/galenholt/hydrogauge

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (18.2%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Basic Info
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 5
  • Releases: 0
Created over 3 years ago · Last pushed over 1 year ago
Metadata Files
Readme Changelog License Citation

README.Rmd 

---
output: 
  github_document:
    df_print: kable
---



```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# hydrogauge



[![R-CMD-check](https://github.com/galenholt/hydrogauge/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/galenholt/hydrogauge/actions/workflows/R-CMD-check.yaml) [![Codecov test coverage](https://codecov.io/gh/galenholt/hydrogauge/graph/badge.svg)](https://app.codecov.io/gh/galenholt/hydrogauge)



## Purpose

hydrogauge is designed to query [Kisters](https://resources.kisters.com.au/public/kisters-web-publishing/) Hydstra (hydllp) and Kisters WISKI/KiWIS water gauge APIs. It does not have complete API coverage, but tries to expose more than many other gauge-pulling packages. Please leave functionality requests in github issues.

## Installation

You can install the development version of hydrogauge from [GitHub](https://github.com/galenholt/hydrogauge) with:

``` r
# install.packages("devtools")
devtools::install_github("galenholt/hydrogauge")
```

## Background

This package was originally conceived to enable a workflow of querying to discover data availability (variables, time spans, locations), followed by pulling data based on those findings. This differs from other, more tailored, approaches to e.g. pulling flow gauges for a specific location, trading off querying ability with ease of use.

hydrogauge attempts to smooth out this tradeoff by providing a set of core functions that give the user access to most of the API functionality using syntax as close to API documentation as possible, as well as a set of wrapper functions that attempt to make the most common use-cases easier and avoid the need to dig into API documentation.

This package was originally designed to query various Australian state water gauge networks and the Australian Bureau of Meteorology (BoM) APIs. Some Australian states use [Kisters](https://resources.kisters.com.au/public/kisters-web-publishing/) Hydstra (hydllp) while the BoM uses Kisters WISKI/KiWIS. Thus, this package is best-tested agains Australian data sources, but is likely to work for many other services use these APIs with a simple change to the URL (argument `portal`).

### API and wrapper functions

The core functions try to stay as close as possible to simple translations of arguments from R to Kisters formats, and of output JSON back to R. This approach lets us get closer to exposing the full functionality and makes for clearer mapping to the [Kisters documentation](https://kisters.com.au/doco/hydllp.htm). The goal here is to expose as much functionality as possible and avoid making too many decisions for the user.

hydrogauge provides convenience functions (primarly `fetch_timeseries`, `fetch_kiwis_timeseries`, and `fetch_hydstra_timeseries`) that automate some of the workflow (such as querying period of record and pulling that range). With the recent inclusion of BoM gauges in WISKi/KiWIS format, it is likely that harmonized wrapper functions will be developed to allow calling both types of databases with the same arguments and returning equivalent outputs.

Caveat: while the functionality provided maps as closely as possible to the underlying API calls, that functionality is not complete. This package does not have complete coverage of the available API calls or their arguments, but is under active development to add missing capabilities. Initial focus has been on identifying what is available to pull and pulling timeseries, along with including new data portals.

### Supported sources

Users can input any url for a KiWIS or HYDSTRA (hydllp) portal, but there are also some that can be called by name (not case sensitive):

-   'vic': the [Victorian water data](https://data.water.vic.gov.au/static.htm) (API url )

-   'nsw' : [New South Wales water data](https://realtimedata.waternsw.com.au/) (API url )

-   'qld': [Queensland water data](https://water-monitoring.information.qld.gov.au/) (API url )

-   'bom': [Australian Bureau of Meteorology](http://www.bom.gov.au/waterdata) (API url )

Other Australian states are in progress and haven't been looked into very thoroughly; for the moment, use BoM as a fallback if you don't know their API urls.

-   [WA](https://wir.water.wa.gov.au/Pages/Water-Information-Reporting.aspx) seems to use HYDSTRA, but the API url hasn't been found yet.

-   Unclear what [SA](https://water.data.sa.gov.au/) is using as a backend, but other packages (e.g. python [mdba-gauge-getter](https://github.com/MDBAuth/MDBA_Gauge_Getter)) use BOM for SA.

-   [Tasmania](https://portal.wrt.tas.gov.au/Data) and [NT](https://ntg.aquaticinformatics.net/Data) have maps that look a lot like BoM

If you're outside Australia, just use your URL. Create an issue (or pull request) to add other named portals.

The [Kisters website](https://resources.kisters.com.au/public/kisters-web-publishing/) provides a list of sites that are likely to work with these functions, but it may take work to find the correct API url, and those not in the list above are untested.

## Useful API documentation

For the base functions that directly access the two APIs, argument names are as close as possible to those used by Kisters. Thus, the API documentation can be very useful in understanding these calls (and identifying not-yet-developed functionality in hydrogauge).

### HYDSTRA

I have tried to keep the function argument names the same as in the API, and the API restricts the options to the function arguments. The API functions are documented by [Kisters](https://kisters.com.au/doco/hydllp.htm) (the creators), and there is a bit more information about options from [Queensland](https://water-monitoring.information.qld.gov.au/wini/Documents/RDMW_API_doco.pdf), though there are discrepancies between states.

### WISKI/KiWIS

Most of the documentation I've found here is from the [Scottish Environment Protection Agency](sepa.org.uk); I have not found good docs straight from Kisters. The [Kisters docs themselves](https://timeseries.sepa.org.uk/KiWIS/KiWIS?datasource=0&service=kisters&type=queryServices&request=getrequestinfo) do seem to be available from SEPA though. These provide information about available functions, allowed variables to request, etc.

A good [SEPA walkthrough](https://timeseriesdoc.sepa.org.uk/api-documentation/api-function-reference/principal-query-functions).

Unlike Hydstra, which provides tailored arguments for different sorts of API control, KiWIS primarily uses text search within columns. This can be more flexible, but means we need to know column names and pay close attention to the regex used to filter those columns to avoid contaminating outputs.

## Similar packages

The [bomWater](https://github.com/buzacott/bomWater) and [kiwisR](https://github.com/rywhale/kiwisR) packages have much of the functionality needed to call the BoM API, targeted at Australia (bomWater) and generally (kiwisR). I have learned a lot from them, and have only chosen to reimplement the BoM work because neither quite had what I needed for my workflow without being convoluted.

There are similar python packages [mdba-gauge-getter](https://github.com/MDBAuth/MDBA_Gauge_Getter), which calls state and BoM, and [bomwater](github.com/csiro-hydroinformatics/pybomwater), which is the BoM interface. Both of these are tailored for flow and stage timeseries, with less emphasis than here on identifying available data and exposing API funcationality.

An obvious missing piece is USGS gauges. The USGS provides the in-house [dataRetrieval](https://github.com/DOI-USGS/dataRetrieval) package. At present, I have not explored wrapping that here and doubt that would make sense, given the existing tool.

## Notes

I have not done much systematic testing of speed for big requests (long periods of record for lots of sites and variables). I am working on optimizing the API calls. It's certainly possible to overflow your memory if you ask for everything for all sites.

## Development plans

This package is under active development, and has been put on github as soon as the main functionality (`get_ts_traces`) was working. The current high-priority next steps are:

-   More helpful wrapper functions that automatically detect and pull data

    -   Especially across both hydstra and KiWIS

-   Selecting and finding sites based on criteria

    -   *Especially* geographic

-   Diagnostic plots for data inspection, though those will in general be made by the user

-   Smarter/faster handling of multiple calls (optimising API calling)

## Contact

Please submit issues on [GitHub](https://github.com/galenholt/hydrogauge/issues)

-   Galen Holt, [g.holt\@deakin.edu.au](mailto:g.holt@deakin.edu.au){.email}

Owner

  • Name: Galen Holt
  • Login: galenholt
  • Kind: user

Citation (CITATION.cff) 

# --------------------------------------------
# CITATION file created with {cffr} R package
# See also: https://docs.ropensci.org/cffr/
# --------------------------------------------
 
cff-version: 1.2.0
message: 'To cite package "hydrogauge" in publications use:'
type: software
license: MIT
title: 'hydrogauge: Pulls information from the Kisters Kiwis and Hydstra APIs'
version: 0.0.0.9000
abstract: This package provides R wrappers to (some) of the API calls possible to
  the Victorian gauge network. I have tried to provide functionality to query available
  options for the different arguments.
authors:
- family-names: Holt
  given-names: Galen
  email: g.holt@deakin.edu.au
  orcid: https://orcid.org/0000-0002-7455-9275
contact:
- family-names: Holt
  given-names: Galen
  email: g.holt@deakin.edu.au
  orcid: https://orcid.org/0000-0002-7455-9275
references:
- type: software
  title: doFuture
  abstract: 'doFuture: Use Foreach to Parallelize via the Future Framework'
  notes: Imports
  url: https://doFuture.futureverse.org
  repository: https://CRAN.R-project.org/package=doFuture
  authors:
  - family-names: Bengtsson
    given-names: Henrik
    email: henrikb@braju.com
    orcid: https://orcid.org/0000-0002-7579-5165
  year: '2024'
  doi: 10.32614/CRAN.package.doFuture
- type: software
  title: dplyr
  abstract: 'dplyr: A Grammar of Data Manipulation'
  notes: Imports
  url: https://dplyr.tidyverse.org
  repository: https://CRAN.R-project.org/package=dplyr
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
    orcid: https://orcid.org/0000-0003-4757-117X
  - family-names: François
    given-names: Romain
    orcid: https://orcid.org/0000-0002-2444-4226
  - family-names: Henry
    given-names: Lionel
  - family-names: Müller
    given-names: Kirill
    orcid: https://orcid.org/0000-0002-1416-3412
  - family-names: Vaughan
    given-names: Davis
    email: davis@posit.co
    orcid: https://orcid.org/0000-0003-4777-038X
  year: '2024'
  doi: 10.32614/CRAN.package.dplyr
- type: software
  title: foreach
  abstract: 'foreach: Provides Foreach Looping Construct'
  notes: Imports
  url: https://github.com/RevolutionAnalytics/foreach
  repository: https://CRAN.R-project.org/package=foreach
  authors:
  - name: Microsoft
  - family-names: Weston
    given-names: Steve
  year: '2024'
  doi: 10.32614/CRAN.package.foreach
- type: software
  title: furrr
  abstract: 'furrr: Apply Mapping Functions in Parallel using Futures'
  notes: Imports
  url: https://furrr.futureverse.org/
  repository: https://CRAN.R-project.org/package=furrr
  authors:
  - family-names: Vaughan
    given-names: Davis
    email: davis@rstudio.com
  - family-names: Dancho
    given-names: Matt
    email: mdancho@business-science.io
  year: '2024'
  doi: 10.32614/CRAN.package.furrr
- type: software
  title: ggplot2
  abstract: 'ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics'
  notes: Imports
  url: https://ggplot2.tidyverse.org
  repository: https://CRAN.R-project.org/package=ggplot2
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
    orcid: https://orcid.org/0000-0003-4757-117X
  - family-names: Chang
    given-names: Winston
    orcid: https://orcid.org/0000-0002-1576-2126
  - family-names: Henry
    given-names: Lionel
  - family-names: Pedersen
    given-names: Thomas Lin
    email: thomas.pedersen@posit.co
    orcid: https://orcid.org/0000-0002-5147-4711
  - family-names: Takahashi
    given-names: Kohske
  - family-names: Wilke
    given-names: Claus
    orcid: https://orcid.org/0000-0002-7470-9261
  - family-names: Woo
    given-names: Kara
    orcid: https://orcid.org/0000-0002-5125-4188
  - family-names: Yutani
    given-names: Hiroaki
    orcid: https://orcid.org/0000-0002-3385-7233
  - family-names: Dunnington
    given-names: Dewey
    orcid: https://orcid.org/0000-0002-9415-4582
  - family-names: Brand
    given-names: Teun
    name-particle: van den
    orcid: https://orcid.org/0000-0002-9335-7468
  year: '2024'
  doi: 10.32614/CRAN.package.ggplot2
- type: software
  title: httr2
  abstract: 'httr2: Perform HTTP Requests and Process the Responses'
  notes: Imports
  url: https://httr2.r-lib.org
  repository: https://CRAN.R-project.org/package=httr2
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@rstudio.com
  year: '2024'
  doi: 10.32614/CRAN.package.httr2
- type: software
  title: lubridate
  abstract: 'lubridate: Make Dealing with Dates a Little Easier'
  notes: Imports
  url: https://lubridate.tidyverse.org
  repository: https://CRAN.R-project.org/package=lubridate
  authors:
  - family-names: Spinu
    given-names: Vitalie
    email: spinuvit@gmail.com
  - family-names: Grolemund
    given-names: Garrett
  - family-names: Wickham
    given-names: Hadley
  year: '2024'
  doi: 10.32614/CRAN.package.lubridate
- type: software
  title: purrr
  abstract: 'purrr: Functional Programming Tools'
  notes: Imports
  url: https://purrr.tidyverse.org/
  repository: https://CRAN.R-project.org/package=purrr
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@rstudio.com
    orcid: https://orcid.org/0000-0003-4757-117X
  - family-names: Henry
    given-names: Lionel
    email: lionel@rstudio.com
  year: '2024'
  doi: 10.32614/CRAN.package.purrr
- type: software
  title: rlang
  abstract: 'rlang: Functions for Base Types and Core R and ''Tidyverse'' Features'
  notes: Imports
  url: https://rlang.r-lib.org
  repository: https://CRAN.R-project.org/package=rlang
  authors:
  - family-names: Henry
    given-names: Lionel
    email: lionel@posit.co
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  year: '2024'
  doi: 10.32614/CRAN.package.rlang
- type: software
  title: stringr
  abstract: 'stringr: Simple, Consistent Wrappers for Common String Operations'
  notes: Imports
  url: https://stringr.tidyverse.org
  repository: https://CRAN.R-project.org/package=stringr
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  year: '2024'
  doi: 10.32614/CRAN.package.stringr
- type: software
  title: tibble
  abstract: 'tibble: Simple Data Frames'
  notes: Imports
  url: https://tibble.tidyverse.org/
  repository: https://CRAN.R-project.org/package=tibble
  authors:
  - family-names: Müller
    given-names: Kirill
    email: kirill@cynkra.com
    orcid: https://orcid.org/0000-0002-1416-3412
  - family-names: Wickham
    given-names: Hadley
    email: hadley@rstudio.com
  year: '2024'
  doi: 10.32614/CRAN.package.tibble
- type: software
  title: tidyr
  abstract: 'tidyr: Tidy Messy Data'
  notes: Imports
  url: https://tidyr.tidyverse.org
  repository: https://CRAN.R-project.org/package=tidyr
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  - family-names: Vaughan
    given-names: Davis
    email: davis@posit.co
  - family-names: Girlich
    given-names: Maximilian
  year: '2024'
  doi: 10.32614/CRAN.package.tidyr
- type: software
  title: tidyselect
  abstract: 'tidyselect: Select from a Set of Strings'
  notes: Imports
  url: https://tidyselect.r-lib.org
  repository: https://CRAN.R-project.org/package=tidyselect
  authors:
  - family-names: Henry
    given-names: Lionel
    email: lionel@posit.co
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  year: '2024'
  doi: 10.32614/CRAN.package.tidyselect
- type: software
  title: future
  abstract: 'future: Unified Parallel and Distributed Processing in R for Everyone'
  notes: Suggests
  url: https://future.futureverse.org
  repository: https://CRAN.R-project.org/package=future
  authors:
  - family-names: Bengtsson
    given-names: Henrik
    email: henrikb@braju.com
    orcid: https://orcid.org/0000-0002-7579-5165
  year: '2024'
  doi: 10.32614/CRAN.package.future
- type: software
  title: knitr
  abstract: 'knitr: A General-Purpose Package for Dynamic Report Generation in R'
  notes: Suggests
  url: https://yihui.org/knitr/
  repository: https://CRAN.R-project.org/package=knitr
  authors:
  - family-names: Xie
    given-names: Yihui
    email: xie@yihui.name
    orcid: https://orcid.org/0000-0003-0645-5666
  year: '2024'
  doi: 10.32614/CRAN.package.knitr
- type: software
  title: rmarkdown
  abstract: 'rmarkdown: Dynamic Documents for R'
  notes: Suggests
  url: https://pkgs.rstudio.com/rmarkdown/
  repository: https://CRAN.R-project.org/package=rmarkdown
  authors:
  - family-names: Allaire
    given-names: JJ
    email: jj@posit.co
  - family-names: Xie
    given-names: Yihui
    email: xie@yihui.name
    orcid: https://orcid.org/0000-0003-0645-5666
  - family-names: Dervieux
    given-names: Christophe
    email: cderv@posit.co
    orcid: https://orcid.org/0000-0003-4474-2498
  - family-names: McPherson
    given-names: Jonathan
    email: jonathan@posit.co
  - family-names: Luraschi
    given-names: Javier
  - family-names: Ushey
    given-names: Kevin
    email: kevin@posit.co
  - family-names: Atkins
    given-names: Aron
    email: aron@posit.co
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  - family-names: Cheng
    given-names: Joe
    email: joe@posit.co
  - family-names: Chang
    given-names: Winston
    email: winston@posit.co
  - family-names: Iannone
    given-names: Richard
    email: rich@posit.co
    orcid: https://orcid.org/0000-0003-3925-190X
  year: '2024'
  doi: 10.32614/CRAN.package.rmarkdown
- type: software
  title: testthat
  abstract: 'testthat: Unit Testing for R'
  notes: Suggests
  url: https://testthat.r-lib.org
  repository: https://CRAN.R-project.org/package=testthat
  authors:
  - family-names: Wickham
    given-names: Hadley
    email: hadley@posit.co
  year: '2024'
  doi: 10.32614/CRAN.package.testthat
  version: '>= 3.0.0'

GitHub Events

Total
  • Issues event: 1
  • Issue comment event: 1
  • Push event: 12
  • Create event: 1
Last Year
  • Issues event: 1
  • Issue comment event: 1
  • Push event: 12
  • Create event: 1

Dependencies

.github/workflows/R-CMD-check.yaml actions
  • actions/checkout v3 composite
  • r-lib/actions/check-r-package v2 composite
  • r-lib/actions/setup-pandoc v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
DESCRIPTION cran
  • dplyr * imports
  • foreach * imports
  • ggplot2 * imports
  • httr2 * imports
  • lubridate * imports
  • purrr * imports
  • rlang * imports
  • stringr * imports
  • tibble * imports
  • tidyr * imports
  • tidyselect * imports
  • doFuture * suggests
  • future * suggests
  • knitr * suggests
  • rmarkdown * suggests
  • testthat >= 3.0.0 suggests