datapackager

An R package to enable reproducible data processing, packaging and sharing.

https://github.com/ropensci/datapackager

Keywords

Repository

An R package to enable reproducible data processing, packaging and sharing.

README.Rmd

---
title: README
output: github_document
bibliography: bibliography.bib
editor_options:
  chunk_output_type: inline
---


```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# DataPackageR

DataPackageR is used to reproducibly process raw data into packaged, analysis-ready data sets.


[![CRAN](https://www.r-pkg.org/badges/version/DataPackageR)]( https://CRAN.R-project.org/package=DataPackageR)
[![R-CMD-check](https://github.com/ropensci/DataPackageR/workflows/R-CMD-check/badge.svg)](https://github.com/ropensci/DataPackageR/actions)
[![Codecov test coverage](https://codecov.io/gh/ropensci/DataPackageR/branch/main/graph/badge.svg)](https://app.codecov.io/gh/ropensci/DataPackageR?branch=main)
[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
[![](https://badges.ropensci.org/230_status.svg)](https://github.com/ropensci/software-review/issues/230)
[![DOI](https://zenodo.org/badge/29267435.svg)](https://doi.org/10.5281/zenodo.1292095)


## Installation

You can install the latest [CRAN](https://cran.r-project.org/package=DataPackageR) release of DataPackageR with:

```{r, eval=FALSE}
install.packages("DataPackageR")
```

You can install the latest development version of DataPackageR from [GitHub](https://github.com/ropensci/DataPackageR) with:

```{r, eval=FALSE}
library(remotes)
remotes::install_github("ropensci/DataPackageR")
```

## What problems does DataPackageR tackle?

You have diverse raw data sets that you need to preprocess and tidy in order to:

- Perform data analysis
- Write a report
- Publish a paper
- Share data with colleagues and collaborators
- Save time in the future when you return to this project but have forgotten all about what you did.

### Why package data sets?

**Definition:** A *data package* is a formal R package whose sole purpose is to contain, access, and / or document data sets.

- **Reproducibility.**

  As described [elsewhere](https://github.com/ropensci/rrrpkg), packaging your data promotes reproducibility.
  R's packaging infrastructure promotes unit testing, documentation, a reproducible build system, and has many other benefits.
  Coopting it for packaging data sets is a natural fit.

- **Collaboration.**

  A data set packaged in R is easy to distribute and share among collaborators, and is easy to install and use.
  All the hard work you've put into documenting and standardizing the tidy data set comes right along with the data package.

- **Documentation.**

  R's package system allows us to document data objects. What's more, the `roxygen2` package makes this very easy to do with [markup tags](https://r-pkgs.org/data.html).
  That documentation is the equivalent of a data dictionary and can be extremely valuable when returning to a project after a period of time.

- **Convenience.**

  Data pre-processing can be time consuming, depending on the data type and raw data sets may be too large to share conveniently in a packaged format.
  Packaging and sharing the small, tidied data saves the users computing time and time spent waiting for downloads.

## Challenges

- **Package size limits.**

  R packages have a 10MB size limit, at least on [CRAN](https://cran.r-project.org/web/packages/policies.html). Bioconductor [ExperimentHub](http://contributions.bioconductor.org/data.html#data) may be able to support larger data packages.

  Sharing large volumes of raw data in an R package format is still not ideal, and there are public biological data repositories better suited for raw data: e.g.,  [GEO](https://www.ncbi.nlm.nih.gov/geo/), [SRA](https://www.ncbi.nlm.nih.gov/sra), [ImmPort](https://www.immport.org/), [ImmuneSpace](https://immunespace.org/), [FlowRepository](http://flowrepository.org/).

  Tools like [datastorr](https://github.com/traitecoevo/datastorr) can help with this and we hope to integrate this into DataPackageR in the future.

- **Manual effort**

  There is still a substantial manual effort to set up the correct directory structures for an R data package. This can dissuade many individuals, particularly new users who have never built an R package, from going this route.

- **Scale**

  Setting up and building R data packages by hand is a workable solution for a small project or a small number of projects, but when dealing with many projects each involving many data sets, tools are needed to help automate the process.

## DataPackageR

DataPackageR provides a number of benefits when packaging your data.

- It aims to automate away much of the tedium of packaging data sets without getting too much in the way, and keeps your processing workflow reproducible.

- It sets up the necessary package structure and files for a data package.

- It allows you to keep the large, raw data and only ship the packaged tidy data, saving space and time consumers of your data set need to spend downloading and re-processing it.

- It maintains a reproducible record (vignettes) of the data processing along with the package. Consumers of the data package can verify how the processing was done, increasing confidence in your data.

- It automates construction of the documentation and maintains a data set version and an md5 fingerprint of each data object in the package. If the data changes and the package is rebuilt, the data version is automatically updated.

## Blog Post - building packages interactively.

See this [rOpenSci blog post](https://ropensci.org/blog/2018/09/18/datapackager/) on how to build data packages interactively using DataPackageR.
This uses several new interfaces: `use_data_object()`, `use_processing_script()` and `use_raw_dataset()` to build up a data package, rather than assuming
the user has all the code and data ready to go for `datapackage_skeleton()`.

## Example

```{r minimal_example}
library(DataPackageR)

# Let's reproducibly package up
# the cars in the mtcars dataset
# with speed > 20.
# Our dataset will be called cars_over_20.
# There are three steps:

# 1. Get the code file that turns the raw data
# into our packaged and processed analysis-ready dataset.
# This is in a file called subsetCars.Rmd located in exdata/tests of the DataPackageR package.
# For your own projects you would write your own Rmd processing file.
processing_code <- system.file(
  "extdata", "tests", "subsetCars.Rmd", package = "DataPackageR"
)

# 2. Create the package framework.
# We pass in the Rmd file in the `processing_code` variable and the names of the data objects it creates (called "cars_over_20")
# The new package is called "mtcars20"
datapackage_skeleton(
  "mtcars20", force = TRUE,
  code_files = processing_code,
  r_object_names = "cars_over_20",
  path = tempdir())

# 3. Run the preprocessing code to build the cars_over_20 data set
# and reproducibly enclose it in the mtcars20 package.
# packageName is the full path to the package source directory created at step 2.
# You'll be prompted for a text description (one line) of the changes you're making.
# These will be added to the NEWS.md file along with the DataVersion in the package source directory.
# If the build is run in non-interactive mode, the description will read
# "Package built in non-interactive mode". You may update it later.
package_build(packageName = file.path(tempdir(),"mtcars20"))

# Update the autogenerated roxygen documentation in data-raw/documentation.R.
# edit(file.path(tempdir(),"mtcars20","R","mtcars20.R"))

# 4. Rebuild the documentation.
document(file.path(tempdir(),"mtcars20"))

# Let's use the package we just created.
# During actual use, the temporary library does not need to be specified.
temp_lib <- file.path(tempdir(),"lib")
dir.create(temp_lib)
install.packages(file.path(tempdir(),"mtcars20_1.0.tar.gz"),
                 type = "source", repos = NULL, lib = temp_lib)
library(mtcars20, lib.loc = temp_lib)
data("cars_over_20") # load the data

# We have our dataset!
# Since we preprocessed it,
# it is clean and under the 5 MB limit for data in packages.
cars_over_20

?cars_over_20 # See the documentation you wrote in data-raw/documentation.R.

# We can easily check the version of the data
data_version("mtcars20")

# You can use an assert to check the data version in  reports and
# analyses that use the packaged data.
assert_data_version(data_package_name = "mtcars20",
                    version_string = "0.1.0",
                    acceptable = "equal")
```

### Reading external data from within R / Rmd processing scripts.

When creating a data package, your processing scripts will need to read your raw data sets in order to process them.
These data sets can be stored in `inst/extdata` of the data package source tree, or elsewhere outside the package source tree.
In order to have portable and reproducible code, you should not use absolute paths to the raw data.
Instead, `DataPackageR` provides several APIs to access the data package project root directory, the `inst/extdata` subdirectory, and the `data` subdirectory.

```{r, eval = FALSE}
# This returns the datapackage source
# root directory.
# In an R or Rmd processing script this can be used to build a path to a directory that is exteral to the package, for
# example if we are dealing with very large data sets where data cannot be packaged.
DataPackageR::project_path()

# This returns the
# inst/extdata directory.
# Raw data sets that are included in the package should be placed there.
# They can be read from that location, which is returned by:
DataPackageR::project_extdata_path()

# This returns the path to the datapackage
# data directory. This can be used to access
# stored data objects already created and saved in `data` from
# other processing scripts.
DataPackageR::project_data_path()
```


## Vignettes

[yaml configuration guide](https://docs.ropensci.org/DataPackageR/articles/YAML_Configuration_Details.html)

[a more detailed technical vignette](https://docs.ropensci.org/DataPackageR/articles/Using_DataPackageR.html)

## Preprint and publication 

The publication describing the package, (Finak *et al.*, 2018), is now available at   [Gates Open Research](https://gatesopenresearch.org/articles/2-31/v2) .


The preprint is on [bioRxiv](https://doi.org/10.1101/342907).

## Similar work

DataPackageR is for processing raw data into tidy data sets and bundling them into R packages. (Note: [datapack](https://github.com/ropensci/datapack) is a **different package** that is used to "create, send and load data from common repositories such as DataONE into the R environment".)


There are a number of tools out there that address similar and complementary problems:

- **datastorr**
  [github repo](https://github.com/traitecoevo/datastorr)

  Simple data retrieval and versioning using GitHub to store data.

    - Caches downloads and uses github releases to version data.
    - Deal consistently with translating the file stored online into a loaded data object
    - Access multiple versions of the data at once

  `datastorrr` could be used with DataPackageR to store / access remote raw data sets, remotely store / access tidied data that are too large to fit in the package itself.

- **fst**
  [github repo](https://github.com/fstpackage/fst)

  `fst` provides lightning fast serialization of data frames.

- **The modern data package**
  [pdf](https://github.com/noamross/2018-04-18-rstats-nyc/blob/master/Noam_Ross_ModernDataPkg_rstatsnyc_2018-04-20.pdf)

  A presentation from \@noamross touching on modern tools for open science and reproducibility. Discusses `datastorr` and `fst` as well as standardized metadata and documentation.

- **rrrpkg**
  [github repo](https://github.com/ropensci/rrrpkg)

  A document from rOpenSci describing using an R package as a research compendium. Based on ideas originally introduced by Robert Gentleman and Duncan Temple Lang (Gentleman and Lang (2004))

- **template**
  [github repo](https://github.com/ropensci/rrrpkg)

  An R package template for data packages.

 See the [publication](#publication) for further discussion.

## Code of conduct

Please note that this project is released with a [Contributor Code of Conduct](https://github.com/ropensci/DataPackageR/blob/main/CODE_OF_CONDUCT.md).
  By participating in this project you agree to abide by its terms.

### References

1. Gentleman, Robert, and Duncan Temple Lang. 2004. “Statistical Analyses and Reproducible Research.” Bioconductor Project Working Papers, Bioconductor project working papers,. bepress.

2. Finak G, Mayer B, Fulp W, et al. DataPackageR: Reproducible data preprocessing, standardization and sharing using R/Bioconductor for collaborative data analysis. Gates Open Res 2018, 2:31
(DOI: 10.12688/gatesopenres.12832.1)



  [![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)

Owner

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "identifier": "DataPackageR",
  "description": "A framework to help construct R data packages in a reproducible manner. Potentially time consuming processing of raw data sets into analysis ready data sets is done in a reproducible manner and decoupled from the usual 'R CMD build' process so that data sets can be processed into R objects in the data package and the data package can then be shared, built, and installed by others without the need to repeat computationally costly data processing. The package maintains data provenance by turning the data processing scripts into package vignettes, as well as enforcing documentation and version checking of included data objects. Data packages can be version controlled on 'GitHub', and used to share data for manuscripts, collaboration and reproducible research.",
  "name": "DataPackageR: Construct Reproducible Analytic Data Sets as R Packages",
  "relatedLink": [
    "https://docs.ropensci.org/DataPackageR/",
    "https://CRAN.R-project.org/package=DataPackageR"
  ],
  "codeRepository": "https://github.com/ropensci/DataPackageR",
  "issueTracker": "https://github.com/ropensci/DataPackageR/issues",
  "license": "https://spdx.org/licenses/MIT",
  "version": "0.16.1",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "R",
    "url": "https://r-project.org"
  },
  "runtimePlatform": "R version 4.4.1 (2024-06-14)",
  "provider": {
    "@id": "https://cran.r-project.org",
    "@type": "Organization",
    "name": "Comprehensive R Archive Network (CRAN)",
    "url": "https://cran.r-project.org"
  },
  "author": [
    {
      "@type": "Person",
      "givenName": "Greg",
      "familyName": "Finak",
      "email": "greg.finak@gmail.com"
    }
  ],
  "contributor": [
    {
      "@type": "Person",
      "givenName": "Paul",
      "familyName": "Obrecht"
    },
    {
      "@type": "Person",
      "givenName": "Ellis",
      "familyName": "Hughes",
      "email": "ellishughes@live.com",
      "@id": "https://orcid.org/0000-0003-0637-4436"
    },
    {
      "@type": "Person",
      "givenName": "Jimmy",
      "familyName": "Fulp",
      "email": "williamjfulp@gmail.com"
    },
    {
      "@type": "Person",
      "givenName": "Marie",
      "familyName": "Vendettuoli",
      "@id": "https://orcid.org/0000-0001-9321-1410"
    },
    {
      "@type": "Person",
      "givenName": "Dave",
      "familyName": "Slager",
      "email": "dslager@fredhutch.org",
      "@id": "https://orcid.org/0000-0003-2525-2039"
    },
    {
      "@type": "Person",
      "givenName": "Jason",
      "familyName": "Taylor",
      "email": "jmtaylor@fredhutch.org"
    }
  ],
  "copyrightHolder": [
    {
      "@type": "Person",
      "givenName": "Greg",
      "familyName": "Finak",
      "email": "greg.finak@gmail.com"
    }
  ],
  "maintainer": [
    {
      "@type": "Person",
      "givenName": "Dave",
      "familyName": "Slager",
      "email": "dslager@fredhutch.org",
      "@id": "https://orcid.org/0000-0003-2525-2039"
    }
  ],
  "softwareSuggestions": [
    {
      "@type": "SoftwareApplication",
      "identifier": "covr",
      "name": "covr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=covr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "data.tree",
      "name": "data.tree",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=data.tree"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "spelling",
      "name": "spelling",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=spelling"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "testthat",
      "name": "testthat",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=testthat"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "withr",
      "name": "withr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=withr"
    }
  ],
  "softwareRequirements": {
    "1": {
      "@type": "SoftwareApplication",
      "identifier": "R",
      "name": "R",
      "version": ">= 3.5.0"
    },
    "2": {
      "@type": "SoftwareApplication",
      "identifier": "cli",
      "name": "cli",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=cli"
    },
    "3": {
      "@type": "SoftwareApplication",
      "identifier": "desc",
      "name": "desc",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=desc"
    },
    "4": {
      "@type": "SoftwareApplication",
      "identifier": "digest",
      "name": "digest",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=digest"
    },
    "5": {
      "@type": "SoftwareApplication",
      "identifier": "futile.logger",
      "name": "futile.logger",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=futile.logger"
    },
    "6": {
      "@type": "SoftwareApplication",
      "identifier": "knitr",
      "name": "knitr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=knitr"
    },
    "7": {
      "@type": "SoftwareApplication",
      "identifier": "pkgbuild",
      "name": "pkgbuild",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=pkgbuild"
    },
    "8": {
      "@type": "SoftwareApplication",
      "identifier": "pkgload",
      "name": "pkgload",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=pkgload"
    },
    "9": {
      "@type": "SoftwareApplication",
      "identifier": "rmarkdown",
      "name": "rmarkdown",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rmarkdown"
    },
    "10": {
      "@type": "SoftwareApplication",
      "identifier": "roxygen2",
      "name": "roxygen2",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=roxygen2"
    },
    "11": {
      "@type": "SoftwareApplication",
      "identifier": "rprojroot",
      "name": "rprojroot",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rprojroot"
    },
    "12": {
      "@type": "SoftwareApplication",
      "identifier": "usethis",
      "name": "usethis",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=usethis"
    },
    "13": {
      "@type": "SoftwareApplication",
      "identifier": "utils",
      "name": "utils"
    },
    "14": {
      "@type": "SoftwareApplication",
      "identifier": "yaml",
      "name": "yaml",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=yaml"
    },
    "SystemRequirements": "pandoc - https://pandoc.org"
  },
  "fileSize": "860.52KB",
  "releaseNotes": "https://github.com/ropensci/DataPackageR/blob/master/NEWS.md",
  "readme": "https://github.com/ropensci/DataPackageR/blob/main/README.md",
  "contIntegration": [
    "https://github.com/ropensci/DataPackageR/actions",
    "https://app.codecov.io/gh/ropensci/DataPackageR?branch=main"
  ],
  "developmentStatus": "https://www.repostatus.org/#active",
  "review": {
    "@type": "Review",
    "url": "https://github.com/ropensci/software-review/issues/230",
    "provider": "https://ropensci.org"
  },
  "keywords": [
    "r",
    "r-package",
    "reproducibility",
    "rstats",
    "peer-reviewed"
  ]
}

GitHub Events

Committers

Last synced: about 2 years ago

Issues and Pull Requests

Last synced: 6 months ago

Packages