Data

Tools

Indexes

Applications

Experiments

Open Source Science

bench

High Precision Timing of R Expressions

https://github.com/r-lib/bench

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (19.0%) to scientific vocabulary

Keywords

r

Keywords from Contributors

setup tidy-data latex rmarkdown data-manipulation visualisation documentation-tool parsing unit-testing grammar
Last synced: 11 months ago · JSON representation

Repository

High Precision Timing of R Expressions

Basic Info
  • Host: GitHub
  • Owner: r-lib
  • License: other
  • Language: R
  • Default Branch: main
  • Homepage: http://bench.r-lib.org/
  • Size: 12.3 MB
Statistics
  • Stars: 253
  • Watchers: 7
  • Forks: 25
  • Open Issues: 15
  • Releases: 8
Topics
r
Created over 8 years ago · Last pushed over 1 year ago
Metadata Files
Readme Changelog License Code of conduct

README.Rmd 

---
output: github_document
---



```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
options(width = 120)
```

# bench



[![CRAN status](https://www.r-pkg.org/badges/version/bench)](https://cran.r-project.org/package=bench)
[![R-CMD-check](https://github.com/r-lib/bench/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/r-lib/bench/actions/workflows/R-CMD-check.yaml)
[![Codecov test coverage](https://codecov.io/gh/r-lib/bench/graph/badge.svg)](https://app.codecov.io/gh/r-lib/bench)



The goal of bench is to benchmark code, tracking execution time, memory allocations and garbage collections.

## Installation

You can install the release version from [CRAN](https://cran.r-project.org/) with:

```{r, eval = FALSE}
install.packages("bench")
```

Or you can install the development version from [GitHub](https://github.com/) with:

``` r
# install.packages("pak")
pak::pak("r-lib/bench")
```

## Features

`bench::mark()` is used to benchmark one or a series of expressions, we feel it has a number of advantages over [alternatives](#alternatives).

-   Always uses the highest precision APIs available for each operating system (often nanoseconds).
-   Tracks memory allocations for each expression.
-   Tracks the number and type of R garbage collections per expression iteration.
-   Verifies equality of expression results by default, to avoid accidentally benchmarking inequivalent code.
-   Has `bench::press()`, which allows you to easily perform and combine benchmarks across a large grid of values.
-   Uses adaptive stopping by default, running each expression for a set amount of time rather than for a specific number of iterations.
-   Expressions are run in batches and summary statistics are calculated after filtering out iterations with garbage collections. This allows you to isolate the performance and effects of garbage collection on running time (for more details see [Neal 2014](https://radfordneal.wordpress.com/2014/02/02/inaccurate-results-from-microbenchmark/)).

The times and memory usage are returned as custom objects which have human readable formatting for display (e.g. `104ns`) and comparisons (e.g. `x$mem_alloc > "10MB"`).

There is also full support for plotting with [ggplot2](https://ggplot2.tidyverse.org/) including custom scales and formatting.

## Usage

### `bench::mark()`

Benchmarks can be run with `bench::mark()`, which takes one or more expressions to benchmark against each other.

```{r example, cache = TRUE}
library(bench)
set.seed(42)

dat <- data.frame(
  x = runif(10000, 1, 1000),
  y = runif(10000, 1, 1000)
)
```

`bench::mark()` will throw an error if the results are not equivalent, so you don't accidentally benchmark inequivalent code.

```{r example-1, error = TRUE, cache = TRUE, dependson = "example"}
bench::mark(
  dat[dat$x > 500, ],
  dat[which(dat$x > 499), ],
  subset(dat, x > 500)
)
```

Results are easy to interpret, with human readable units.

```{r example-2, cache = TRUE, dependson = "example"}
bnch <- bench::mark(
  dat[dat$x > 500, ],
  dat[which(dat$x > 500), ],
  subset(dat, x > 500)
)
bnch
```

By default the summary uses absolute measures, however relative results can be obtained by using `relative = TRUE` in your call to `bench::mark()` or calling `summary(relative = TRUE)` on the results.

```{r example-3, cache = TRUE, dependson = "example-2"}
summary(bnch, relative = TRUE)
```

### `bench::press()`

`bench::press()` is used to run benchmarks against a grid of parameters.
Provide setup and benchmarking code as a single unnamed argument then define sets of values as named arguments.
The full combination of values will be expanded and the benchmarks are then *pressed* together in the result.
This allows you to benchmark a set of expressions across a wide variety of input sizes, perform replications and other useful tasks.

```{r example2, cache = TRUE}
set.seed(42)

create_df <- function(rows, cols) {
  out <- replicate(cols, runif(rows, 1, 100), simplify = FALSE)
  out <- setNames(out, rep_len(c("x", letters), cols))
  as.data.frame(out)
}

results <- bench::press(
  rows = c(1000, 10000),
  cols = c(2, 10),
  {
    dat <- create_df(rows, cols)
    bench::mark(
      min_iterations = 100,
      bracket = dat[dat$x > 500, ],
      which = dat[which(dat$x > 500), ],
      subset = subset(dat, x > 500)
    )
  }
)

results
```

## Plotting

`ggplot2::autoplot()` can be used to generate an informative default plot.
This plot is colored by gc level (0, 1, or 2) and faceted by parameters (if any).
By default it generates a [beeswarm](https://github.com/eclarke/ggbeeswarm#geom_quasirandom) plot, however you can also specify other plot types (`jitter`, `ridge`, `boxplot`, `violin`).
See `?autoplot.bench_mark` for full details.

```{r autoplot, message = FALSE, warning = FALSE, cache = TRUE, dependson = "example2", dpi = 300}
ggplot2::autoplot(results)
```

You can also produce fully custom plots by un-nesting the results and working with the data directly.

```{r custom-plot, message = FALSE, cache = TRUE, dependson = "example2", dpi = 300}
library(tidyverse)

results %>%
  unnest(c(time, gc)) %>%
  filter(gc == "none") %>%
  mutate(expression = as.character(expression)) %>%
  ggplot(aes(x = mem_alloc, y = time, color = expression)) +
  geom_point() +
  scale_color_bench_expr(scales::brewer_pal(type = "qual", palette = 3))
```

## `system_time()`

**bench** also includes `system_time()`, a higher precision alternative to [system.time()](https://www.rdocumentation.org/packages/base/versions/3.5.0/topics/system.time).

```{r system-time, cache = TRUE}
bench::system_time({
  i <- 1
  while (i < 1e7) {
    i <- i + 1
  }
})

bench::system_time(Sys.sleep(.5))
```

## Alternatives {#alternatives}

-   [rbenchmark](https://cran.r-project.org/package=rbenchmark)
-   [microbenchmark](https://cran.r-project.org/package=microbenchmark)
-   [tictoc](https://cran.r-project.org/package=tictoc)
-   [system.time()](https://www.rdocumentation.org/packages/base/versions/3.5.0/topics/system.time)

Owner

  • Name: R infrastructure
  • Login: r-lib
  • Kind: organization

GitHub Events

Total
  • Create event: 1
  • Release event: 1
  • Issues event: 8
  • Watch event: 8
  • Delete event: 1
  • Issue comment event: 2
  • Push event: 27
  • Pull request event: 17
Last Year
  • Create event: 1
  • Release event: 1
  • Issues event: 8
  • Watch event: 8
  • Delete event: 1
  • Issue comment event: 2
  • Push event: 27
  • Pull request event: 17

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 347
  • Total Committers: 20
  • Avg Commits per committer: 17.35
  • Development Distribution Score (DDS): 0.262
Past Year
  • Commits: 20
  • Committers: 4
  • Avg Commits per committer: 5.0
  • Development Distribution Score (DDS): 0.2
Top Committers
Name Email Commits
Jim Hester j****r@g****m 256
DavisVaughan d****s@r****m 63
Lionel Henry l****y@g****m 4
John Blischak j****k@g****m 3
Kirill Müller k****r@m****g 3
Michael Chirico c****m@g****m 2
Mara Averick m****k@g****m 2
olivroy 5****y 2
Andrew Manderson a****n@l****k 1
David C Hall d****l 1
Enrico Spinielli e****i@g****m 1
HughParsonage h****e@g****m 1
James J Balamuta c****s 1
Lorenz Walthert l****t@i****m 1
Matthias Grenié m****e@e****r 1
Paul Liétar l****p@g****m 1
Robin Lovelace R****e 1
Romain François r****n@p****t 1
Tanner Stauss 4****s 1
oranwutang 6****g 1
Committer Domains (Top 20 + Academic)
purrple.cat: 1 ens-lyon.fr: 1 live.co.uk: 1 google.com: 1 mailbox.org: 1 rstudio.com: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 78
  • Total pull requests: 47
  • Average time to close issues: 5 months
  • Average time to close pull requests: 3 months
  • Total issue authors: 46
  • Total pull request authors: 23
  • Average comments per issue: 1.44
  • Average comments per pull request: 0.68
  • Merged pull requests: 38
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 3
  • Pull requests: 14
  • Average time to close issues: 21 days
  • Average time to close pull requests: about 2 hours
  • Issue authors: 2
  • Pull request authors: 3
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.07
  • Merged pull requests: 12
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • jimhester (10)
  • DavisVaughan (9)
  • hadley (4)
  • HughParsonage (3)
  • Anirban166 (3)
  • Robinlovelace (3)
  • multimeric (2)
  • simonpcouch (2)
  • jennybc (2)
  • MatthieuStigler (2)
  • lorenzwalthert (2)
  • romainfrancois (2)
  • moldach (1)
  • DavZim (1)
  • EmilRehnberg (1)
Pull Request Authors
  • DavisVaughan (20)
  • olivroy (4)
  • MichaelChirico (3)
  • plietar (2)
  • batpigandme (2)
  • simonpcouch (2)
  • lionel- (1)
  • fabian-s (1)
  • hhau (1)
  • jdblischak (1)
  • oranwutang (1)
  • Zopolis4 (1)
  • Anirban166 (1)
  • krlmlr (1)
  • lorenzwalthert (1)
Top Labels
Issue Labels
feature (13) upkeep (3) reprex (2) breaking change :skull_and_crossbones: (1)
Pull Request Labels

Packages

  • Total packages: 3
  • Total downloads:
    • cran 59,243 last-month
  • Total docker downloads: 2,743
  • Total dependent packages: 45
    (may contain duplicates)
  • Total dependent repositories: 142
    (may contain duplicates)
  • Total versions: 26
  • Total maintainers: 1
cran.r-project.org: bench

High Precision Timing of R Expressions

  • Versions: 10
  • Dependent Packages: 45
  • Dependent Repositories: 141
  • Downloads: 59,243 Last month
  • Docker Downloads: 2,743
Rankings
Dependent repos count: 1.7%
Stargazers count: 1.8%
Dependent packages count: 2.0%
Forks count: 3.2%
Downloads: 3.7%
Average: 5.4%
Docker downloads count: 20.2%
Maintainers (1)
davis@posit.co
Last synced: 11 months ago
proxy.golang.org: github.com/r-lib/bench
  • Versions: 9
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.4%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 11 months ago
conda-forge.org: r-bench
  • Versions: 7
  • Dependent Packages: 0
  • Dependent Repositories: 1
Rankings
Dependent repos count: 24.3%
Stargazers count: 24.9%
Average: 33.7%
Forks count: 34.1%
Dependent packages count: 51.6%
Last synced: 11 months ago

Dependencies

DESCRIPTION cran
  • R >= 3.1 depends
  • glue * imports
  • methods * imports
  • pillar * imports
  • profmem * imports
  • rlang >= 0.2.0 imports
  • stats * imports
  • tibble >= 3.0.1 imports
  • utils * imports
  • covr * suggests
  • dplyr * suggests
  • forcats * suggests
  • ggbeeswarm * suggests
  • ggplot2 * suggests
  • ggridges * suggests
  • jsonlite * suggests
  • mockery * suggests
  • parallel * suggests
  • scales * suggests
  • testthat * suggests
  • tidyr >= 0.8.1 suggests
  • vctrs * suggests
  • withr * suggests
.github/workflows/R-CMD-check.yaml actions
  • actions/checkout v3 composite
  • r-lib/actions/check-r-package v2 composite
  • r-lib/actions/setup-pandoc v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
.github/workflows/pkgdown.yaml actions
  • JamesIves/github-pages-deploy-action v4.4.1 composite
  • actions/checkout v3 composite
  • r-lib/actions/setup-pandoc v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
.github/workflows/pr-commands.yaml actions
  • actions/checkout v3 composite
  • r-lib/actions/pr-fetch v2 composite
  • r-lib/actions/pr-push v2 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite
.github/workflows/test-coverage.yaml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v3 composite
  • r-lib/actions/setup-r v2 composite
  • r-lib/actions/setup-r-dependencies v2 composite