autotest

Automatic testing of R packages

https://github.com/ropensci-review-tools/autotest

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (20.0%) to scientific vocabulary

Keywords

automated-testing fuzzing r testing

Last synced: 4 months ago · JSON representation

Repository

Automatic testing of R packages

Basic Info

Host: GitHub
Owner: ropensci-review-tools
Language: R
Default Branch: main
Homepage: https://docs.ropensci.org/autotest
Size: 1.75 MB

Statistics

Stars: 54
Watchers: 2
Forks: 4
Open Issues: 23
Releases: 0

Topics

automated-testing fuzzing r testing

Created over 5 years ago · Last pushed 4 months ago

Metadata Files

Readme Codemeta

README.Rmd

---
title: "autotest"
output:
  md_document:
    variant: gfm

  rmarkdown::html_vignette:
    self_contained: no
---

# autotest 



```{r setup, include=FALSE}
knitr::opts_chunk$set (
    collapse = TRUE,
    warning = TRUE,
    message = TRUE,
    width = 120,
    comment = "#>",
    fig.retina = 2,
    fig.path = "README-"
)
```



[![R build
status](https://github.com/ropensci-review-tools/autotest/workflows/R-CMD-check/badge.svg)](https://github.com/ropensci-review-tools/autotest/actions?query=workflow%3AR-CMD-check)
[![codecov](https://codecov.io/gh/ropensci-review-tools/autotest/branch/master/graph/badge.svg)](https://codecov.io/gh/ropensci-review-tools/autotest)
[![Project Status:
Concept](https://www.repostatus.org/badges/latest/concept.svg)](https://www.repostatus.org/#concept)



Automatic mutation testing of R packages. Mutation in the sense of mutating
inputs (parameters) to function calls. `autotest` primarily works by scraping
documented examples for all functions, and mutating the parameters input to
those functions.


## Installation

The easiest way to install this package is via the associated
[`r-universe`](https://ropensci-review-tools.r-universe.dev/ui#builds). As
shown there, simply enable the universe with

```{r options, eval = FALSE}
options (repos = c (
    ropenscireviewtools = "https://ropensci-review-tools.r-universe.dev",
    CRAN = "https://cloud.r-project.org"
))
```

And then install the usual way with,

```{r install, eval = FALSE}
install.packages ("autotest")
```

Alternatively, the package can be installed by running one of the following
lines:

```{r gh-installation, eval = FALSE}
# install.packages("remotes")
remotes::install_git ("https://git.sr.ht/~mpadge/autotest")
remotes::install_bitbucket ("mpadge/autotest")
remotes::install_gitlab ("mpadge/autotest")
remotes::install_github ("ropensci-review-tools/autotest")
```

The package can then be loaded the usual way:
```{r load-fakey, eval = FALSE}
library (autotest)
```
```{r load, echo = FALSE, message = FALSE}
devtools::load_all (".", export_all = FALSE)
```

## Usage

The simply way to use the package is

```{r autotest-example, eval = FALSE}
x <- autotest_package ("")
```

The main argument to the [`autotest_package()`
function](https://docs.ropensci.org/autotest/reference/autotest_package.html)
can either be the name of an installed package, or a path to a local directory
containing the source for a package. The result is a `data.frame` of errors,
warnings, and other diagnostic messages issued during package `auotest`-ing.
The function has an additional parameter, `functions`, to restrict tests to
specified functions only.

By default,
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
returns a list of all tests applied to a package without actually running them.
To implement those tests, set the parameter `test` to `TRUE`. Results are only returned for tests 
in which functions do not behave as expected, whether through triggering
errors, warnings, or other behaviour as described below. The ideal behaviour of
`autotest_package()` is to return nothing (or strictly, `NULL`), indicating
that all tests passed successfully. See the [main package
vignette](https://docs.ropensci.org/autotest/articles/autotest.html) for an
introductory tour of the package.

## What is tested?

The package includes a function which lists all tests currently implemented.

```{r autotest_types}
autotest_types ()
```

That functions returns a [`tibble`](https://tibble.tidyverse.org) describing
`r nrow(autotest_types())` unique tests. The default behaviour of
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
with `test = FALSE` uses these test types to identify which tests will be
applied to each parameter and function. The table returned from
[`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html)
can be used to selectively switch tests off by setting values in the `test`
column to `FALSE`, as demonstrated below.

## How Does It Work?

The package works by scraping documented examples from all `.Rd` help files,
and using those to identify the types of all parameters to all functions. Usage
therefore first requires that the usage of all parameters be demonstrated in
example code.

As described above, tests can also be selectively applied to particular
functions through the parameters `functions`, used to nominate functions to
include in tests, or `exclude`, used to nominate functions to exclude from
tests. The following code illustrates.

```{r stats-var-no-test, fig.show = "hide"}
x <- autotest_package (package = "stats", functions = "var", test = FALSE)
print (x)
```

Testing the `var` function also tests `cor` and `cov`, because these are all
documented within a single `.Rd` help file. Typing `?var` shows that the help
topic is `cor`, and that the examples include the three functions, `var`,
`cor`, and `cov`. That result details the `r nrow (x)` tests which would be
applied to the `var` function from the `stats` package. These `r nrow (x)`
tests yield the following results when actually applied:

```{r stats-var-test}
y <- autotest_package (package = "stats", functions = "var", test = TRUE)
print (y)
```

And only `r nrow (y)` of the original `r nrow (x)` tests produced unexpected
behaviour. There were in fact only `r length (unique (y$operation))` kinds of
tests which produced these `r nrow (y)` results:


```{r unique-operations}
unique (y$operation)
```

One of these involves conversion of a vector to a list-column representation
(via `I(as.list())`). Relatively few packages accept this kind of input,
even though doing so is relatively straightforward. The following lines
demonstrate how these tests can be switched off when `autotest`-ing a package.
The `autotest_types()` function, used above to extract information on all types
of tests, also accepts a single argument listing the `test_name` entries of any
tests which are to be switched off.

```{r stats-var-test-switch}
types <- autotest_types (notest = "vector_to_list_col")
y <- autotest_package (
    package = "stats", functions = "var",
    test = TRUE, test_data = types
)
print (y)
```

Those tests are still returned from `autotest_package()`, but with `test =
FALSE` to indicate they were not run, and a `type` of "no_test" rather than the
previous "diagnostic".


## Can `autotest` automatically create tests in my `tests` directory?

Not yet, but that should be possible soon. In the meantime, there are
[`testthat`](https://testthat.r-lib.org) expectations, listed in the [main
package
functions](https://docs.ropensci.org/autotest/reference/index.html),
which enable `autotest` to be used in a package's test suite.


## Prior work

1. The
   [`great-expectations`](https://github.com/great-expectations/great_expectations)
   framework for python, described in [this medium
   article](https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a).
2. [`QuickCheck`](https://hackage.haskell.org/package/QuickCheck) for Haskell
3. [`mutate`](https://github.com/mbj/mutant) for ruby
4. [`mutant`](https://github.com/ropensci/mutant) for mutation of R code itself

## Code of Conduct

Please note that this package is released with a [Contributor Code of
Conduct](https://ropensci.org/code-of-conduct/). By contributing to this
project, you agree to abide by its terms.

## Contributors






All contributions to this project are gratefully acknowledged using the [`allcontributors` package](https://github.com/ropenscilabs/allcontributors) following the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!

### Code









mpadge






helske






maelle






simpar1471






### Issue Authors









noamross






njtierney






JeffreyRStevens






bbolker






mattfidler






kieranjmartin






statnmap










vgherard






christophsax






joelnitta






santikka






gilbertocamara






### Issue Contributors









schneiderpy

Owner

Name: ropensci-review-tools
Login: ropensci-review-tools
Kind: organization

Website: https://ropensci-review-tools.readthedocs.io/
Repositories: 10
Profile: https://github.com/ropensci-review-tools

Tools for automation of software review at rOpenSci

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "identifier": "autotest",
  "description": "Automatic testing of R packages via a simple YAML schema.",
  "name": "autotest: Automatic Package Testing",
  "relatedLink": "https://docs.ropensci.org/autotest/",
  "codeRepository": "https://github.com/ropensci-review-tools/autotest",
  "issueTracker": "https://github.com/ropensci-review-tools/autotest/issues",
  "license": "https://spdx.org/licenses/GPL-3.0",
  "version": "0.0.2.215",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "R",
    "url": "https://r-project.org"
  },
  "runtimePlatform": "R version 4.2.1 (2022-06-23)",
  "author": [
    {
      "@type": "Person",
      "givenName": "Mark",
      "familyName": "Padgham",
      "email": "mark.padgham@email.com",
      "@id": "https://orcid.org/0000-0003-2172-5265"
    }
  ],
  "contributor": [
    {
      "@type": "Person",
      "givenName": "Jouni",
      "familyName": "Helske",
      "@id": "https://orcid.org/0000-0001-7130-793X"
    }
  ],
  "maintainer": [
    {
      "@type": "Person",
      "givenName": "Mark",
      "familyName": "Padgham",
      "email": "mark.padgham@email.com",
      "@id": "https://orcid.org/0000-0003-2172-5265"
    }
  ],
  "softwareSuggestions": [
    {
      "@type": "SoftwareApplication",
      "identifier": "devtools",
      "name": "devtools",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=devtools"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "dplyr",
      "name": "dplyr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=dplyr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "DT",
      "name": "DT",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=DT"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "fs",
      "name": "fs",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=fs"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "knitr",
      "name": "knitr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=knitr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "rmarkdown",
      "name": "rmarkdown",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rmarkdown"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "roxygen2",
      "name": "roxygen2",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=roxygen2"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "tidyr",
      "name": "tidyr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=tidyr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "usethis",
      "name": "usethis",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=usethis"
    }
  ],
  "softwareRequirements": {
    "1": {
      "@type": "SoftwareApplication",
      "identifier": "cli",
      "name": "cli",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=cli"
    },
    "2": {
      "@type": "SoftwareApplication",
      "identifier": "data.table",
      "name": "data.table",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=data.table"
    },
    "3": {
      "@type": "SoftwareApplication",
      "identifier": "digest",
      "name": "digest",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=digest"
    },
    "4": {
      "@type": "SoftwareApplication",
      "identifier": "here",
      "name": "here",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=here"
    },
    "5": {
      "@type": "SoftwareApplication",
      "identifier": "magrittr",
      "name": "magrittr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=magrittr"
    },
    "6": {
      "@type": "SoftwareApplication",
      "identifier": "memoise",
      "name": "memoise",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=memoise"
    },
    "7": {
      "@type": "SoftwareApplication",
      "identifier": "methods",
      "name": "methods"
    },
    "8": {
      "@type": "SoftwareApplication",
      "identifier": "pkgload",
      "name": "pkgload",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=pkgload"
    },
    "9": {
      "@type": "SoftwareApplication",
      "identifier": "rlang",
      "name": "rlang",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rlang"
    },
    "10": {
      "@type": "SoftwareApplication",
      "identifier": "testthat",
      "name": "testthat",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=testthat"
    },
    "11": {
      "@type": "SoftwareApplication",
      "identifier": "tibble",
      "name": "tibble",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=tibble"
    },
    "12": {
      "@type": "SoftwareApplication",
      "identifier": "withr",
      "name": "withr",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=withr"
    },
    "13": {
      "@type": "SoftwareApplication",
      "identifier": "yaml",
      "name": "yaml",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=yaml"
    },
    "SystemRequirements": {}
  },
  "fileSize": "570.294KB",
  "readme": "https://github.com/ropensci-review-tools/autotest/blob/main/README.md",
  "contIntegration": [
    "https://github.com/ropensci-review-tools/autotest/actions?query=workflow%3AR-CMD-check",
    "https://codecov.io/gh/ropensci-review-tools/autotest"
  ],
  "developmentStatus": "https://www.repostatus.org/#concept"
}

GitHub Events

Total

Issues event: 6
Issue comment event: 5
Push event: 2

Last Year

Issues event: 6
Issue comment event: 5
Push event: 2

Issues and Pull Requests

Last synced: 4 months ago

All Time

Total issues: 5
Total pull requests: 0
Average time to close issues: 1 day
Average time to close pull requests: N/A
Total issue authors: 4
Total pull request authors: 0
Average comments per issue: 0.4
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 5
Pull requests: 0
Average time to close issues: 1 day
Average time to close pull requests: N/A
Issue authors: 4
Pull request authors: 0
Average comments per issue: 0.4
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

schneiderpy (2)
joelnitta (1)
abigailkeller (1)
marcburri (1)
simpar1471 (1)
KlausVigo (1)
vincenzocoia (1)
mpadge (1)

Pull Request Authors

simpar1471 (1)

Top Labels

Issue Labels

Pull Request Labels

Dependencies

DESCRIPTION cran

cli * imports
data.table * imports
digest * imports
here * imports
magrittr * imports
memoise * imports
methods * imports
pkgload * imports
rlang * imports
testthat * imports
tibble * imports
withr * imports
yaml * imports
DT * suggests
devtools * suggests
dplyr * suggests
fs * suggests
knitr * suggests
rmarkdown * suggests
roxygen2 * suggests
tidyr * suggests
usethis * suggests

.github/workflows/check-standard.yaml actions

actions/checkout v2 composite
r-lib/actions/check-r-package v2 composite
r-lib/actions/setup-pandoc v2 composite
r-lib/actions/setup-r v2 composite
r-lib/actions/setup-r-dependencies v2 composite

.github/workflows/test-coverage.yaml actions

actions/checkout v2 composite
r-lib/actions/setup-r v2 composite
r-lib/actions/setup-r-dependencies v2 composite

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science