Data

Tools

Indexes

Applications

Experiments

Open Source Science

Managing Larger Data on a GitHub Repository

Managing Larger Data on a GitHub Repository - Published in JOSS (2018)

https://github.com/ropensci/piggyback

Science Score: 95.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 4 DOI reference(s) in README and JOSS metadata
  • ✓
    Academic publication links
    Links to: joss.theoj.org, zenodo.org
  • ✓
    Committers with academic emails
    3 of 18 committers (16.7%) from academic institutions
  • ○
    Institutional organization owner
  • ✓
    JOSS paper metadata
    Published in Journal of Open Source Software

Keywords

data-store git-lfs peer-reviewed r r-package rstats

Keywords from Contributors

codemeta json-ld tidyverse weather-data air-pollution air-pollution-levels air-quality defra
Last synced: 4 months ago · JSON representation

Repository

:package: for using large(r) data files on GitHub

Basic Info
Statistics
  • Stars: 194
  • Watchers: 7
  • Forks: 27
  • Open Issues: 3
  • Releases: 17
Topics
data-store git-lfs peer-reviewed r r-package rstats
Created over 7 years ago · Last pushed 8 months ago
Metadata Files
Readme Changelog License Code of conduct Codemeta

README.Rmd 

---
output: github_document
---




```{r setup, include = FALSE}
knitr::opts_chunk$set(
  eval = FALSE,
  collapse = TRUE,
  message = FALSE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# piggyback 

  

[![lifecycle](https://img.shields.io/badge/lifecycle-stable-green.svg)](https://lifecycle.r-lib.org/articles/stages.html)
[![R-CMD-check](https://github.com/ropensci/piggyback/workflows/R-CMD-check/badge.svg)](https://github.com/ropensci/piggyback/actions)
[![Coverage status](https://codecov.io/gh/ropensci/piggyback/branch/master/graph/badge.svg)](https://app.codecov.io/github/ropensci/piggyback?branch=master)
[![CRAN status](https://www.r-pkg.org/badges/version/piggyback)](https://cran.r-project.org/package=piggyback)
[![Peer Review Status](https://badges.ropensci.org/220_status.svg)](https://github.com/ropensci/software-review/issues/220)
[![DOI](https://zenodo.org/badge/132979724.svg)](https://zenodo.org/badge/latestdoi/132979724)
[![DOI](http://joss.theoj.org/papers/10.21105/joss.00971/status.svg)](https://doi.org/10.21105/joss.00971)
  

`piggyback` provides an R interface for storing files as GitHub release assets,
which is a convenient way for large/binary data files to _piggyback_ onto public 
and private GitHub repositories. This package includes functions for file downloads,
uploads, and managing releases, which then are passed to the GitHub API. 

No authentication is required to download data from public repositories.

## Installation
Install from CRAN via:
```{r}
install.packages("piggyback")
```
You can install the development version from [GitHub](https://github.com/ropensci/piggyback)
with either r-universe or with remotes:
```{r}
install.packages("piggyback", repos = c('https://ropensci.r-universe.dev', getOption("repos")))
# install.packages("remotes")
remotes::install_github("ropensci/piggyback")
```
## Usage
See [getting started vignette](https://docs.ropensci.org/piggyback/articles/piggyback.html)
for a more comprehensive introduction.

Download data attached to a GitHub release:
```{r}
library(piggyback)
pb_download("iris2.tsv.gz", 
            repo = "cboettig/piggyback-tests",
            tag = "v0.0.1",
            dest = tempdir())
#> ℹ Downloading "iris2.tsv.gz"...
#> |======================================================| 100%
fs::dir_tree(tempdir())
#> /tmp/RtmpWxJSZj
#> └── iris2.tsv.gz
```
Downloading from private repos or uploading to any repo requires authentication, 
specifically a GitHub Personal Access Token (PAT). This can be stored as a
[gh::gh_token()](https://usethis.r-lib.org/articles/git-credentials.html#get-a-personal-access-token-pat)
or a GITHUB_PAT environment variable - for more information, see the vignette notes on
[authentication](https://docs.ropensci.org/piggyback/articles/piggyback.html#authentication).

We can also upload data to a release. Start by creating a release:
```{r}
pb_release_create(repo = "cboettig/piggyback-tests", tag = "v0.0.2")
#> ✔ Created new release "v0.0.2".
```
then upload to it:
```{r}
readr::write_tsv(mtcars, "mtcars.tsv.gz")
pb_upload("mtcars.tsv.gz", repo = "cboettig/piggyback-tests")
#> ℹ Uploading to latest release: "v0.0.2".
#> ℹ Uploading mtcars.tsv.gz ...
#> |===================================================| 100%
```

For improved performance, we can also use piggyback files with 
[cloud native](https://docs.ropensci.org/piggyback/articles/cloud_native.html)
workflows to query data without downloading it first. 

## Motivations

A brief video overview presented as part of Tan Ho's [RStudioConf2022 talk](https://www.youtube.com/watch?v=wzcz4xNGeTI&t=655s):

https://github.com/ropensci/piggyback/assets/38083823/a1dff640-1bba-4c06-bad2-feda34f47387

`piggyback` allows you to store data alongside your repository as release assets,
which helps you:

- store files larger than 50MB
- bypass the 2GB GitHub repo size limit

- avoid the [downsides](https://archive.is/3D16r) of Git LFS
- version data flexibly (by creating/uploading to a new release)
- work with public and private repositories, **for free**

For more about motivations, see this discussion of
[alternatives](https://docs.ropensci.org/piggyback/articles/alternatives.html).

## Contributing

Please note that this project is released with a 
[Contributor Code of Conduct](https://ropensci.org/code-of-conduct/).
By participating in this project you agree to abide by its terms.

```{r eval=TRUE, include=FALSE}
codemeta::write_codemeta()
```

[![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)

Owner

  • Name: rOpenSci
  • Login: ropensci
  • Kind: organization
  • Email: info@ropensci.org
  • Location: Berkeley, CA

JOSS Publication

Managing Larger Data on a GitHub Repository
Published
September 24, 2018
DOI
Volume 3, Issue 29, Page 971
Authors
Carl Boettiger ORCID
University of California, Berkeley
Editor
Arfon Smith ORCID
Tags
data GitHub
View PDF Review Thread Software Archive

CodeMeta (codemeta.json) 

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "identifier": "piggyback",
  "description": "Helps store files as GitHub release assets, which is a convenient way for large/binary data files to piggyback onto public and private GitHub repositories. Includes functions for file downloads, uploads, and managing releases via the GitHub API.",
  "name": "piggyback: Managing Larger Data on a GitHub Repository",
  "codeRepository": "https://github.com/ropensci/piggyback",
  "issueTracker": "https://github.com/ropensci/piggyback/issues",
  "license": "https://spdx.org/licenses/GPL-3.0",
  "version": "0.1.5.9004",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "R",
    "url": "https://r-project.org"
  },
  "runtimePlatform": "R version 4.3.2 (2023-10-31)",
  "provider": {
    "@id": "https://cran.r-project.org",
    "@type": "Organization",
    "name": "Comprehensive R Archive Network (CRAN)",
    "url": "https://cran.r-project.org"
  },
  "author": [
    {
      "@type": "Person",
      "givenName": "Carl",
      "familyName": "Boettiger",
      "email": "cboettig@gmail.com",
      "@id": "https://orcid.org/0000-0002-1642-628X"
    },
    {
      "@type": "Person",
      "givenName": "Tan",
      "familyName": "Ho",
      "@id": "https://orcid.org/0000-0001-8388-5155"
    }
  ],
  "contributor": [
    {
      "@type": "Person",
      "givenName": "Mark",
      "familyName": "Padgham",
      "@id": "https://orcid.org/0000-0003-2172-5265"
    },
    {
      "@type": "Person",
      "givenName": [
        "Jeffrey",
        "O"
      ],
      "familyName": "Hanson",
      "@id": "https://orcid.org/0000-0002-4716-6134"
    },
    {
      "@type": "Person",
      "givenName": "Kevin",
      "familyName": "Kuo",
      "@id": "https://orcid.org/0000-0001-7803-7901"
    }
  ],
  "copyrightHolder": [
    {
      "@type": "Person",
      "givenName": "Carl",
      "familyName": "Boettiger",
      "email": "cboettig@gmail.com",
      "@id": "https://orcid.org/0000-0002-1642-628X"
    }
  ],
  "maintainer": [
    {
      "@type": "Person",
      "givenName": "Carl",
      "familyName": "Boettiger",
      "email": "cboettig@gmail.com",
      "@id": "https://orcid.org/0000-0002-1642-628X"
    }
  ],
  "softwareSuggestions": [
    {
      "@type": "SoftwareApplication",
      "identifier": "arrow",
      "name": "arrow",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=arrow"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "spelling",
      "name": "spelling",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=spelling"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "readr",
      "name": "readr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=readr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "covr",
      "name": "covr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=covr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "testthat",
      "name": "testthat",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=testthat"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "knitr",
      "name": "knitr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=knitr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "rmarkdown",
      "name": "rmarkdown",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rmarkdown"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "gert",
      "name": "gert",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=gert"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "withr",
      "name": "withr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=withr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "magrittr",
      "name": "magrittr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=magrittr"
    }
  ],
  "softwareRequirements": {
    "1": {
      "@type": "SoftwareApplication",
      "identifier": "cli",
      "name": "cli",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=cli"
    },
    "2": {
      "@type": "SoftwareApplication",
      "identifier": "glue",
      "name": "glue",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=glue"
    },
    "3": {
      "@type": "SoftwareApplication",
      "identifier": "gh",
      "name": "gh",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=gh"
    },
    "4": {
      "@type": "SoftwareApplication",
      "identifier": "httr",
      "name": "httr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=httr"
    },
    "5": {
      "@type": "SoftwareApplication",
      "identifier": "jsonlite",
      "name": "jsonlite",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=jsonlite"
    },
    "6": {
      "@type": "SoftwareApplication",
      "identifier": "fs",
      "name": "fs",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=fs"
    },
    "7": {
      "@type": "SoftwareApplication",
      "identifier": "memoise",
      "name": "memoise",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=memoise"
    },
    "8": {
      "@type": "SoftwareApplication",
      "identifier": "rlang",
      "name": "rlang",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rlang"
    },
    "SystemRequirements": null
  },
  "fileSize": "397.705KB"
}

GitHub Events

Total
  • Create event: 2
  • Commit comment event: 1
  • Issues event: 2
  • Watch event: 6
  • Delete event: 1
  • Issue comment event: 8
  • Push event: 5
  • Pull request review event: 1
  • Pull request event: 2
Last Year
  • Create event: 2
  • Commit comment event: 1
  • Issues event: 2
  • Watch event: 6
  • Delete event: 1
  • Issue comment event: 8
  • Push event: 5
  • Pull request review event: 1
  • Pull request event: 2

Committers

Last synced: 5 months ago

All Time
  • Total Commits: 317
  • Total Committers: 18
  • Avg Commits per committer: 17.611
  • Development Distribution Score (DDS): 0.382
Past Year
  • Commits: 4
  • Committers: 2
  • Avg Commits per committer: 2.0
  • Development Distribution Score (DDS): 0.25
Top Committers
Name Email Commits
Carl Boettiger c****g@g****m 196
Tan-DESKPC t****n@t****a 91
Carl Boettiger c****g@g****m 6
edavidaja d****d@r****m 3
jeffreyhanson j****n@u****u 3
mrcaseb r****e@g****m 3
Kevin Kuo k****o@g****m 2
Anna Krystalli a****i@g****m 2
Laurent Drouet l****t@g****m 2
Andrea Gilardi a****i@u****t 1
Christophe Dervieux c****x@g****m 1
James Balamuta b****2@i****u 1
Jeroen Ooms j****s@g****m 1
Pat Schloss p****s@u****u 1
Philippe Massicotte p****e@h****m 1
Stefanie Butland s****d@g****m 1
rOpenSci Bot m****t@g****m 1
olivroy 5****y 1
Committer Domains (Top 20 + Academic)
umich.edu: 1 illinois.edu: 1 unimib.it: 1 uqconnect.edu.au: 1 rstudio.com: 1 tanho.ca: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 67
  • Total pull requests: 60
  • Average time to close issues: 3 months
  • Average time to close pull requests: 8 days
  • Total issue authors: 24
  • Total pull request authors: 17
  • Average comments per issue: 2.4
  • Average comments per pull request: 1.25
  • Merged pull requests: 56
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 3
  • Average time to close issues: 3 months
  • Average time to close pull requests: about 2 hours
  • Issue authors: 1
  • Pull request authors: 2
  • Average comments per issue: 5.0
  • Average comments per pull request: 1.0
  • Merged pull requests: 3
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • tanho63 (21)
  • cboettig (10)
  • mpadge (4)
  • kevinykuo (3)
  • mrcaseb (3)
  • 1beb (3)
  • PMassicotte (2)
  • Skourtis (2)
  • bschilder (2)
  • mbirkett-liv (1)
  • Robinlovelace (1)
  • moldach (1)
  • raymondben (1)
  • karinorman (1)
  • MiguelRodo (1)
Pull Request Authors
  • tanho63 (29)
  • cboettig (12)
  • olivroy (2)
  • kevinykuo (2)
  • pschloss (2)
  • mrcaseb (2)
  • annakrystalli (2)
  • edavidaja (1)
  • cderv (1)
  • stefaniebutland (1)
  • agila5 (1)
  • 1beb (1)
  • lolow (1)
  • jeffreyhanson (1)
  • PMassicotte (1)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 4
  • Total downloads:
    • cran 119,090 last-month
  • Total docker downloads: 21,640
  • Total dependent packages: 5
    (may contain duplicates)
  • Total dependent repositories: 30
    (may contain duplicates)
  • Total versions: 39
  • Total maintainers: 2
proxy.golang.org: github.com/ropensci/piggyback
  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.4%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 4 months ago
cran.r-project.org: piggyback

Managing Larger Data on a GitHub Repository

  • Versions: 11
  • Dependent Packages: 4
  • Dependent Repositories: 30
  • Downloads: 119,090 Last month
  • Docker Downloads: 21,640
Rankings
Stargazers count: 2.5%
Forks count: 2.8%
Dependent repos count: 5.0%
Average: 9.6%
Dependent packages count: 10.9%
Downloads: 11.8%
Docker downloads count: 24.9%
Maintainers (1)
cboettig@gmail.com
Last synced: 4 months ago
spack.io: r-piggyback

Managing Larger Data on a GitHub Repository Because larger (> 50 MB) data files cannot easily be committed to git, a different approach is required to manage data associated with an analysis in a GitHub repository. This package provides a simple work-around by allowing larger (up to 2 GB) data files to piggyback on a repository as assets attached to individual GitHub releases. These files are not handled by git in any way, but instead are uploaded, downloaded, or edited directly by calls through the GitHub API. These data files can be versioned manually by creating different releases. This approach works equally well with public or private repositories. Data can be uploaded and downloaded programmatically from scripts. No authentication is required to download data from public repositories.

  • Versions: 11
  • Dependent Packages: 1
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Stargazers count: 15.8%
Forks count: 19.2%
Average: 22.9%
Dependent packages count: 56.4%
Maintainers (1)
jgaeb
Last synced: 4 months ago
conda-forge.org: r-piggyback
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Stargazers count: 25.8%
Forks count: 30.3%
Dependent repos count: 34.0%
Average: 35.3%
Dependent packages count: 51.2%
Last synced: 4 months ago

Dependencies

DESCRIPTION cran
  • cli * imports
  • fs * imports
  • gh * imports
  • glue * imports
  • httr * imports
  • jsonlite * imports
  • lubridate * imports
  • memoise * imports
  • covr * suggests
  • datasets * suggests
  • gert * suggests
  • knitr * suggests
  • magrittr * suggests
  • readr * suggests
  • rmarkdown * suggests
  • spelling * suggests
  • testthat * suggests
  • usethis * suggests
.github/workflows/R-CMD-check.yaml actions
  • actions/checkout v3 composite
  • r-lib/actions/check-r-package v2 composite
  • r-lib/actions/setup-pandoc v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
.github/workflows/test-coverage.yaml actions
  • actions/checkout v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite