Data

Tools

Indexes

Applications

Experiments

Open Source Science

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

https://github.com/sosy-lab/benchexec

Science Score: 59.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 21 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ✓
    Committers with academic emails
    30 of 184 committers (16.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.6%) to scientific vocabulary

Keywords

benchmark benchmark-framework benchmarking cgroups linux python resource-measurement

Keywords from Contributors

legislative-texts programming-language
Last synced: 6 months ago · JSON representation

Repository

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

Basic Info
  • Host: GitHub
  • Owner: sosy-lab
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 27.3 MB
Statistics
  • Stars: 269
  • Watchers: 15
  • Forks: 211
  • Open Issues: 87
  • Releases: 66
Topics
benchmark benchmark-framework benchmarking cgroups linux python resource-measurement
Created about 11 years ago · Last pushed 6 months ago
Metadata Files
Readme Changelog License Zenodo

README.md

BenchExec

A Framework for Reliable Benchmarking and Resource Measurement

Build Status Apache 2.0 License PyPI version DOI

News and Updates: - Taito Ohsumi got accepted with their project for adding optional timestamps to logs for participation in Google Summer of Code! Congratulations and thanks to Google for providing the opportunity! - BenchExec 3.25 can use fuse-overlayfs to make container mode work out-of-the-box again! - BenchExec 3.18 brings support for systems with cgroups v2! - We now provide an Ubuntu PPA that makes installing and upgrading BenchExec easier (docs). - An extended version of our paper on BenchExec and its background was published as open access in the journal STTT, you can read Reliable Benchmarking: Requirements and Solutions online. We also provide a set of overview slides.

To help new or inexperienced users get started with reliable benchmarking right away, we offer a quickstart guide that contains a brief explanation of the issues of common setups as well as the (few) steps necessary to setup and use BenchExec instead.

BenchExec is a framework for reliable benchmarking and resource measurement and provides a standalone solution for benchmarking that takes care of important low-level details for accurate, precise, and reproducible measurements as well as result handling and analysis for large sets of benchmark runs. However, even users of other benchmarking frameworks or scripts can benefit from BenchExec by letting it perform the resource measurements and limits instead of less reliable tools and techniques like time or ulimit, and results produced by BenchExec can easily be exported for use with other tools.

In particular, BenchExec provides three major features:

  • execution of arbitrary commands with precise and reliable measurement and limitation of resource usage (e.g., CPU time and memory), and isolation against other running processes
    (provided by runexec, a replacement for time and similar tools)
  • an easy way to define benchmarks with specific tool configurations and resource limits, and automatically executing them on large sets of input files
    (provided by benchexec on top of runexec)
  • generation of interactive tables and plots for the results
    (provided by table-generator for results produced with benchexec)

Unlike other benchmarking frameworks, BenchExec is able to reliably measure and limit resource usage of the benchmarked tool even if the latter spawns subprocesses. In order to achieve this, it uses the cgroups feature of the Linux kernel to correctly handle groups of processes. For proper isolation of the benchmarks, it uses (if available) Linux user namespaces and an overlay filesystem (either kernel-based or fuse-overlayfs) to create a container that restricts interference of the executed tool with the benchmarking host. More information on why this is necessary and the problems with other tools can be found in our paper Reliable Benchmarking: Requirements and Solutions (open access) and our slides (starting with slide "Checklist").

BenchExec is intended for benchmarking non-interactive tools on Linux systems. It measures CPU time, wall time, and memory usage of a tool, and allows to specify limits for these resources. It also allows to limit the CPU cores and (on NUMA systems) memory regions, and the container mode allows to restrict filesystem and network access. In addition to measuring resource usage, BenchExec can optionally verify that the result of the tool was as expected and extract further statistical data from the output. Results from multiple runs can be combined into CSV and interactive HTML tables, of which the latter provide scatter and quantile plots (have a look at our demo table).

BenchExec works only on Linux and needs a one-time setup of cgroups by the machine's administrator. The actual benchmarking can be done by any user and does not need root access.

BenchExec was originally developed for use with the software verification framework CPAchecker and is now developed as an independent project at the Software Systems Lab of the Ludwig-Maximilians-Universität München (LMU Munich).

Links

Literature

License and Copyright

BenchExec is licensed under the Apache 2.0 License, copyright Dirk Beyer. Exceptions are some tool-info modules and third-party code that is bundled in the HTML tables, which are available under several other free licenses (cf. folder LICENSES).

Authors

Maintainer: Philipp Wendler

Contributors: - Eshaan Aggarwal - Aditya Arora - Levente Bajczi - Dirk Beyer - Laura Bschor - Thomas Bunk - Montgomery Carter - Po-Chun Chien - Andreas Donig - Florian Eder - Karlheinz Friedberger - Robin Gloster - Sam Grayson - Peter Häring - Florian Heck - Chinmay Joshi - George Karpenkov - Mike Kazantsev - Hugo van Kemenade - Tobias Kleinert - Michael Lachner - Thomas Lemberger - Lorenz Leutgeb - Sebastian Ott - Stefan Löwe - Stephan Lukasczyk - Tobias Meggendorfer - Alexander von Rhein - Alexander Schremmer - Dennis Simon - Andreas Stahlbauer - Thomas Stieglmaier - Martin Yankov - Hojan Young - Ilja Zakharov - and lots of more people who integrated tools into BenchExec

Users of BenchExec

Several well-known international competitions use BenchExec, such as SMT-COMP, SV-COMP (software verification), the Termination Competition, and Test-Comp. In particular in SV-COMP BenchExec was used successfully for benchmarking in all instances of the competition and with a wide variety of benchmarked tools and millions of benchmark runs per year. BenchExec is also integrated into the cluster-based logic-solving service StarExec (GitHub).

The developers of the following tools use BenchExec:

If you would like to be listed here, contact us.

Owner

  • Name: SoSy-Lab
  • Login: sosy-lab
  • Kind: organization
  • Location: LMU Munich, Germany

GitHub Events

Total
  • Create event: 28
  • Commit comment event: 2
  • Release event: 5
  • Issues event: 37
  • Watch event: 44
  • Delete event: 25
  • Issue comment event: 144
  • Push event: 152
  • Pull request event: 124
  • Pull request review comment event: 240
  • Pull request review event: 245
  • Fork event: 19
Last Year
  • Create event: 28
  • Commit comment event: 2
  • Release event: 5
  • Issues event: 37
  • Watch event: 44
  • Delete event: 25
  • Issue comment event: 144
  • Push event: 152
  • Pull request event: 124
  • Pull request review comment event: 240
  • Pull request review event: 245
  • Fork event: 19

Committers

Last synced: about 2 years ago

All Time
  • Total Commits: 4,499
  • Total Committers: 184
  • Avg Commits per committer: 24.451
  • Development Distribution Score (DDS): 0.512
Past Year
  • Commits: 290
  • Committers: 30
  • Avg Commits per committer: 9.667
  • Development Distribution Score (DDS): 0.445
Top Committers
Name Email Commits
Philipp Wendler u****i@p****e 2,195
Dennis Simon d****4@g****m 266
Dirk Beyer D****r@s****g 184
lachnerm m****r@o****e 165
Dirk Beyer d****r 94
Tobias Kleinert t****t@g****e 93
Sudeep Kanav s****v@l****e 93
Johan Paulsson j****n@c****e 76
Laura Bschor l****r@n****m 75
Thomas Lemberger l****h@g****m 73
Daniel Dietsch d****h@g****m 72
Robin Gloster m****l@g****n 59
Thomas Lemberger l****7@s****e 58
Karlheinz Friedberger k****r@g****m 52
Thomas Bunk T****9@g****m 43
Johan j****n@c****m 42
alohamora a****7@g****m 38
Thomas Lemberger l****e@f****e 37
Thomas Lemberger s****i@g****m 30
Martin Spiessl g****t@2****e 30
SvenUmbricht S****t@g****m 29
Po-Chun Chien p****n@s****e 22
Gidon Ernst g****t@s****e 21
Matthias Kettl m****l@s****e 15
Williame Rocha w****0@g****m 13
thaprau 4****u 12
Martin Spiessl 3****l 12
George Karpenkov g****e@m****e 12
Zvonimir z****r@c****u 12
Daniel Dietsch d****h@i****e 11
and 154 more...
Committer Domains (Top 20 + Academic)
sosy.ifi.lmu.de: 6 qq.com: 5 fi.muni.cz: 4 mail.muni.cz: 3 fim.uni-passau.de: 3 cs.rwth-aachen.de: 2 mq.edu.au: 2 informatik.uni-freiburg.de: 2 cs.utah.edu: 2 ispras.ru: 2 163.com: 2 gmx.de: 2 ust.hk: 1 mails.tsinghua.edu.cn: 1 stieglmaier.me: 1 honzamrazek.cz: 1 mails.ucas.ac.cn: 1 debian.org: 1 lukasczyk.me: 1 imperial.ac.uk: 1 virginia.edu: 1 stud.uni-passau.de: 1 informatik.rwth-aachen.de: 1 uni-bremen.de: 1 inria.fr: 1 comp.nus.edu.sg: 1 umn.edu: 1 kth.se: 1 rwth-aachen.de: 1 cs.tu-dortmund.de: 1 iith.ac.in: 1 postgrad.manchester.ac.uk: 1 church.cims.nyu.edu: 1 cs.ox.ac.uk: 1 cmu.edu: 1 berkeley.edu: 1 qmul.ac.uk: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 139
  • Total pull requests: 342
  • Average time to close issues: about 1 year
  • Average time to close pull requests: 28 days
  • Total issue authors: 40
  • Total pull request authors: 69
  • Average comments per issue: 2.96
  • Average comments per pull request: 2.47
  • Merged pull requests: 241
  • Bot issues: 0
  • Bot pull requests: 22
Past Year
  • Issues: 20
  • Pull requests: 140
  • Average time to close issues: 1 day
  • Average time to close pull requests: 5 days
  • Issue authors: 10
  • Pull request authors: 31
  • Average comments per issue: 1.4
  • Average comments per pull request: 1.06
  • Merged pull requests: 104
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • PhilippWendler (64)
  • dbeyer (10)
  • incaseoftrouble (8)
  • charmoniumQ (4)
  • sim642 (4)
  • Mariem5005 (3)
  • rmonat (3)
  • EshaanAgg (3)
  • igouy (3)
  • younghojan (2)
  • JawHawk (2)
  • ssardina (2)
  • onderjan (2)
  • leventeBajczi (2)
  • gipsyh (2)
Pull Request Authors
  • Po-Chun-Chien (28)
  • marian-lingsch (24)
  • dependabot[bot] (22)
  • dbeyer (18)
  • leventeBajczi (18)
  • ricffb (14)
  • EshaanAgg (12)
  • PhilippWendler (11)
  • younghojan (10)
  • JawHawk (9)
  • sim642 (8)
  • t0hsumi (8)
  • Mamatha1718 (7)
  • SakshiAgrawal18 (7)
  • incaseoftrouble (7)
Top Labels
Issue Labels
container (24) cgroups (20) HTML table (20) enhancement (17) low priority (14) GSoC (13) question (12) bug (11) usability (9) resource allocation (4) help wanted (4) duplicate (2) invalid (2) tool support (2) high priority (2) awaiting reply (1)
Pull Request Labels
tool support (162) enhancement (24) dependencies (22) HTML table (22) invalid (10) documentation (10) container (10) javascript (8) cgroups (4) resource allocation (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,583 last-month
  • Total dependent packages: 1
  • Total dependent repositories: 13
  • Total versions: 63
  • Total maintainers: 2
pypi.org: benchexec

A Framework for Reliable Benchmarking and Resource Measurement.

  • Versions: 63
  • Dependent Packages: 1
  • Dependent Repositories: 13
  • Downloads: 1,583 Last month
Rankings
Dependent packages count: 3.2%
Forks count: 3.9%
Dependent repos count: 4.1%
Stargazers count: 5.2%
Average: 5.2%
Downloads: 9.8%
Maintainers (2)
dbeyer PhilippWendler
Last synced: 6 months ago

Dependencies

benchexec/tablegenerator/react-table/package-lock.json npm
  • 1296 dependencies
benchexec/tablegenerator/react-table/package.json npm
  • enzyme ^3.11.0 development
  • enzyme-adapter-react-16 ^1.15.6 development
  • eslint-config-prettier ^8.5.0 development
  • eslint-plugin-prettier ^4.2.1 development
  • license-checker ^25.0.1 development
  • prettier ^2.7.1 development
  • react-app-rewired ^2.2.1 development
  • react-scripts 5.0.1 development
  • react-test-renderer ^17.0.2 development
  • sass ^1.53.0 development
  • @babel/runtime ^7.18.9
  • @fortawesome/fontawesome-svg-core ^6.1.1
  • @fortawesome/free-regular-svg-icons ^6.1.1
  • @fortawesome/free-solid-svg-icons ^6.1.1
  • @fortawesome/react-fontawesome ^0.2.0
  • @zip.js/zip.js ~2.4.26
  • classnames ^2.3.1
  • copy-to-clipboard ^3.3.1
  • deep-equal ^2.0.5
  • path-browserify ^1.0.1
  • rc-slider ^9.7.5
  • react ^17.0.2
  • react-dom ^17.0.2
  • react-modal ^3.15.1
  • react-router-dom ^5.3.3
  • react-table ^7.8.0
  • react-table-sticky ^1.1.3
  • react-vis ^1.11.7
  • regression ^2.0.1
  • setimmediate ^1.0.5
  • yaml ^1.10.2
contrib/p4/docker_files/basic_node/Dockerfile docker
  • ubuntu 18.04 build
contrib/p4/docker_files/ptf_tester/Dockerfile docker
  • ubuntu 18.04 build
contrib/p4/docker_files/switch_bmv2/Dockerfile docker
  • ubuntu 16.04 build