Science Score: 67.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 7 DOI reference(s) in README -
✓Academic publication links
Links to: arxiv.org, zenodo.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.9%) to scientific vocabulary
Keywords
Repository
Execute and document benchmarks reproducibly.
Basic Info
Statistics
- Stars: 92
- Watchers: 6
- Forks: 25
- Open Issues: 48
- Releases: 12
Topics
Metadata Files
README.md
ReBench: Execute and Document Benchmarks Reproducibly
ReBench is a tool to run and document benchmark experiments. Currently, it is often used for benchmarking language implementations, but it can be used to monitor the performance of all kinds of other applications and programs, too.
The ReBench configuration format is a text format based on YAML. A configuration file defines how to build and execute a set of experiments, i.e. benchmarks. It describes which executable was used, which parameters were given to the benchmarks, and the number of iterations to be used to obtain statistically reliable results.
With this approach, the configuration contains all benchmark-specific information to reproduce a benchmark run. However, it does not capture the whole system.
The data of all benchmark runs is recorded in a data file for later analysis. Important for long-running experiments, benchmarks can be aborted and continued at a later time.
ReBench focuses on the execution aspect and does not provide advanced analysis facilities itself. Instead, the recorded results should be processed by dedicated tools such as scripts for statistical analysis in R, Python, etc, or ReBenchDB, for continuous performance tracking.
The documentation for ReBench is hosted at https://rebench.readthedocs.io/.
Goals and Features
ReBench is designed to
- enable reproduction of experiments;
- document all benchmark parameters;
- provide a flexible execution model, with support for interrupting and continuing benchmarking;
- enable the definition of complex sets of comparisons and their flexible execution;
- report results to continuous performance monitoring systems, e.g., ReBenchDB;
- provide basic support for building/compiling benchmarks/experiments on demand;
- be extensible to parse output of custom benchmark harnesses.
ReBench Denoise
Denoise configures a Linux system for benchmarking. It adapts parameters of the CPU frequency management and task scheduling to reduce some of the variability that can cause widely different benchmark results for the same experiment.
Denoise is inspired by Krun, which has many more features to carefully minimize possible interference. Krun is the tool of choice if the most reliable results are required. ReBench only adapts a subset of the parameters, while staying self-contained and minimizing external dependencies.
Non-Goals
ReBench isn't
- a framework for (micro)benchmarks. Instead, it relies on existing harnesses and can be extended to parse their output.
- a performance analysis tool. It is meant to execute experiments and record the corresponding measurements.
- a data analysis tool. It provides only a bare minimum of statistics, but has an easily parseable data format that can be processed, e.g., with R.
Installation
ReBench is implemented in Python and can be installed via pip:
bash
pip install rebench
To reduce noise generated by the system, rebench-denoise depends on:
sudorights.rebenchwill attempt to determine suitable configuration parameters and suggest them. This includes allowing the execution ofrebench-denoiseviasudowithout password and with the permission to set environment variables (SETENV).cpusetto reserve cores for benchmarking. On Ubuntu:apt install cpuset
rebench-denoise is currently tested on Ubuntu and Rocky Linux. It is designed to degrade
gracefully and report the expected implications when it cannot adapt system
settings. See the docs for details.
Usage
A minimal configuration file looks like this:
```yaml
this run definition will be chosen if no parameters are given to rebench
defaultexperiment: all defaultdata_file: 'example.data'
a set of suites with different benchmarks and possibly different settings
benchmarksuites: ExampleSuite: gaugeadapter: RebenchLog command: Harness %(benchmark)s %(input)s %(variable)s inputsizes: [2, 10] variablevalues: - val1 benchmarks: - Bench1 - Bench2
a set of executables for the benchmark execution
executors: MyBin1: path: bin executable: test-vm1.py %(cores)s cores: 1 MyBin2: path: bin executable: test-vm2.py
combining benchmark suites and executions
experiments: Example: suites: - ExampleSuite executions: - MyBin1 - MyBin2 ```
Saved as test.conf, this configuration could be executed with ReBench as follows:
bash
rebench test.conf
See the documentation for details: https://rebench.readthedocs.io/.
Support and Contributions
In case you encounter issues, please feel free to open an issue so that we can help.
For contributions, we use pull requests. For larger contributions, it is likely useful to discuss them upfront in an issue first.
Development Setup
For the development setup, the currently recommended
way is to use pip install --editable . in the root directory of the repository.
You may also want to use a virtual environment to avoid conflicts with other Python packages.
For instance:
bash
git clone https://github.com/smarr/rebench.git
cd rebench
pip install --editable .
Unit tests and linting can be run with:
bash
python -m pytest
python -m pylint rebench
Use in Academia
If you use ReBench for research and in academic publications, please consider citing it.
The preferred citation is:
bibtex
@misc{ReBench:2025,
author = {Marr, Stefan},
doi = {10.5281/zenodo.1311762},
month = {February},
note = {Version 1.3},
publisher = {GitHub},
title = {ReBench: Execute and Document Benchmarks Reproducibly},
year = 2025
}
Some publications that have been using ReBench include:
- Transient Typechecks are (Almost) Free, Roberts et al. 2019.
- Efficient and Deterministic Record & Replay for Actor Languages, D. Aumayr et al. 2018.
- Fully Reflective Execution Environments: Virtual Machines for More Flexible Software, G. Chari et al. 2018.
- Building efficient and highly run-time adaptable virtual machines, G. Chari et al. 2017.
- Improving live debugging of concurrent threads through thread histories, M. Leske et al. 2017.
- Adaptive Just-in-time Value Class Optimization for Lowering Memory Consumption and Improving Execution Time Performance T. Pape et al. 2016
- Cross-Language Compiler Benchmarking---Are We Fast Yet?, S. Marr et al. 2016.
- Tracing vs. Partial Evaluation: Comparing Meta-Compilation Approaches for Self-Optimizing Interpreters, S. Marr, S. Ducasse. 2015.
- Zero-Overhead Metaprogramming: Reflection and Metaobject Protocols Fast and without Compromises, S. Marr et al. 2015.
- Pycket: a tracing JIT for a functional language, S. Bauman et al. 2015.
- Adaptive just-in-time value class optimization: transparent data structure inlining for fast execution, T. Pape et al. 2015.
- Meta-tracing makes a fast Racket, C. F. Bolz et al. 2014.
- Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors, J. Swalens et al. 2013.
- Identifying A Unifying Mechanism for the Implementation of Concurrency Abstractions on Multi-Language Virtual Machines, S. Marr, T. D'Hondt. 2012.
- Insertion Tree Phasers: Efficient and Scalable Barrier Synchronization for Fine-grained Parallelism, S. Marr et al. 2011.
Owner
- Name: Stefan Marr
- Login: smarr
- Kind: user
- Location: Canterbury, UK
- Company: University of Kent
- Website: https://www.stefan-marr.de/
- Twitter: smarr
- Repositories: 140
- Profile: https://github.com/smarr
Investigating concurrency, language implementation, and VM technology.
Citation (CITATION.cff)
cff-version: 1.2.0
title: 'ReBench: Execute and Document Benchmarks Reproducibly'
message: >-
If you use this software, please consider citing. Either
based on the metadata found in this file, or the main DOI:
https://doi.org/10.5281/zenodo.1311762
type: software
authors:
- given-names: Stefan
family-names: Marr
email: s.marr@kent.ac.uk
affiliation: University of Kent
orcid: 'https://orcid.org/0000-0001-9059-5180'
identifiers:
- type: doi
value: 10.5281/zenodo.1311762
description: Release and archival on Zenodo
repository-code: 'https://github.com/smarr/rebench'
url: 'https://rebench.dev/'
repository-artifact: 'https://doi.org/10.5281/zenodo.1311762'
abstract: >-
ReBench is a tool to run and document benchmark
experiments. Currently, it is mostly used for benchmarking
language implementations, but it can be used to monitor
the performance of all kinds of other applications and
programs, too.
keywords:
- science
- benchmarking
- performance tracking
- continuous benchmarking
- continuous testing
- research
- reproducibility
license: MIT
GitHub Events
Total
- Create event: 18
- Release event: 1
- Issues event: 12
- Watch event: 11
- Delete event: 19
- Issue comment event: 28
- Push event: 112
- Pull request review comment event: 1
- Pull request review event: 2
- Pull request event: 34
Last Year
- Create event: 18
- Release event: 1
- Issues event: 12
- Watch event: 11
- Delete event: 19
- Issue comment event: 28
- Push event: 112
- Pull request review comment event: 1
- Pull request review event: 2
- Pull request event: 34
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Stefan Marr | g****t@s****e | 942 |
| naomiGrew | 9****w | 16 |
| Octave Larose | o****e@h****r | 10 |
| Max Leske | m****e@g****m | 4 |
| Isaac | I****r@l****u | 3 |
| Corey McCandless | c****s@a****m | 3 |
| Ben Orchard | t****n@g****m | 3 |
| kyvyt0n | z****r@g****m | 2 |
| ad1622 | d****2@g****m | 2 |
| Lode Hoste | l****e@n****m | 2 |
| Guido Chari | c****g@g****m | 2 |
| Torbjörn Gannholm | t****e@M****l | 1 |
| Edd Barrett | v****1@g****m | 1 |
| Fabio Niephaus | c****e@f****m | 1 |
| Martin McClure | m****e@g****m | 1 |
| Tobias Pape | t****s@n****e | 1 |
| Yi Lin | q****n@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 78
- Total pull requests: 105
- Average time to close issues: 12 months
- Average time to close pull requests: 23 days
- Total issue authors: 11
- Total pull request authors: 11
- Average comments per issue: 1.37
- Average comments per pull request: 1.93
- Merged pull requests: 99
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 8
- Pull requests: 33
- Average time to close issues: 12 days
- Average time to close pull requests: 1 day
- Issue authors: 3
- Pull request authors: 1
- Average comments per issue: 0.75
- Average comments per pull request: 1.15
- Merged pull requests: 33
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- smarr (64)
- vext01 (4)
- krono (2)
- daumayr (1)
- qinsoon (1)
- tobega (1)
- raehik (1)
- o- (1)
- clin1234 (1)
- OctaveLarose (1)
- cajomferro (1)
Pull Request Authors
- smarr (106)
- naomiGrew (6)
- antonzhukovin (3)
- raehik (3)
- krono (2)
- martinmcclure (2)
- qinsoon (1)
- cmccandless (1)
- tobega (1)
- vext01 (1)
- fniephaus (1)
- OctaveLarose (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 2,135 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 20
- Total maintainers: 1
pypi.org: rebench
Execute and document benchmarks reproducibly.
- Homepage: https://github.com/smarr/ReBench
- Documentation: https://rebench.readthedocs.io/
- License: MIT
-
Latest release: 1.3.0
published about 1 year ago
Rankings
Maintainers (1)
Dependencies
- PyYAML >=3.12
- humanfriendly >=8.0
- psutil >=5.6.7
- py-cpuinfo ==7.0.0
- pykwalify_version ,
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/checkout v2 composite
- actions/setup-python v1 composite
- mkdocs >=1.4.2
- mkdocs-gitbook ==0.0.1