benchpark

An open collaborative repository for reproducible specifications of HPC benchmarks and cross site benchmarking environments

https://github.com/llnl/benchpark

Keywords

Repository

An open collaborative repository for reproducible specifications of HPC benchmarks and cross site benchmarking environments

README.rst

.. raw:: html

    

      

        
        
        
        
        
      
    

.. image:: https://codecov.io/gh/llnl/benchpark/branch/develop/graph/badge.svg?token=BKYKTSQ8DK
    :target: https://codecov.io/gh/llnl/benchpark

.. image:: https://github.com/llnl/benchpark/workflows/ci/badge.svg
    :target: https://github.com/LLNL/benchpark/actions/workflows/ci.yml

.. image:: https://github.com/llnl/benchpark/workflows/nightly/badge.svg
    :target: https://github.com/LLNL/benchpark/actions/workflows/nightly.yml

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/psf/black

.. image:: https://img.shields.io/badge/Dashboard-CDash-blue
    :target: https://my.cdash.org/index.php?project=Benchpark

Overview
--------
You can find detailed documentation at `software.llnl.gov/benchpark
`_. Benchpark can also be found on `GitHub
`_.

Benchpark is an open collaborative repository for reproducible specifications of HPC benchmarks.
Benchpark enables cross-site collaboration on benchmarking by providing a mechanism for sharing
reproducible, working specifications for the following:

1. **System Specifications** (benchmark and experiment agnostic)

* Hardware information
* System software environment information (available compilers, MPI)
* Scheduler and launcher

2. **Benchmark Specifications** (system and experiment agnostic)

* Source repo and version
* Build configuration (with `Spack `_)
* Run configuration (with `Ramble `_)

3. **Experiment Specifications** (specific benchmark experiment on a system specification)

* Programming model (e.g., OpenMP, CUDA, ROCm) for the benchmark on a given system
* Parameters for individual runs in a study

Dependencies
------------
Benchpark uses the following open source projects for specifying configurations:

* `Spack `_ for building benchmark and dependencies
* `Ramble `_ for run configurations

Community
---------
Benchpark is an open source project.  Questions, discussion,and contributions of
new specifications as well as updates and improvements to existing specifications are welcome.

* `Adding a System Specification `_
* `Adding a Benchmark Specification `_
* `Adding an Experiment Specification `_

We use `github discussions `_ for Q&A and discussion.

Contributing
------------
To contribute to Benchpark, please open a `pull request
`_
to the `develop` branch.  Your PR must pass Benchpark's unit tests, and must be `PEP 8 `_ compliant.

Authors and citations
---------------------
Many thanks to Benchpark's `contributors `_.

Benchpark was created by Olga Pearce, Alec Scott, Greg Becker, Riyaz Haque, and Nathan Hanford.

To cite Benchpark, please use the following citation:

Olga Pearce, Alec Scott, Gregory Becker, Riyaz Haque, Nathan Hanford, Stephanie Brink,
Doug Jacobsen, Heidi Poxon, Jens Domke, and Todd Gamblin. 2023.
Towards Collaborative Continuous Benchmarking for HPC.
In Workshops of The International Conference on High Performance Computing,
Network, Storage, and Analysis (SC-W 2023), November 12–17, 2023, Denver, CO, USA.
ACM, New York, NY, USA, 9 pages.
`doi.org/10.1145/3624062.3624135 `_.

Olga Pearce, Gregory Becker, Stephanie Brink, Nathan Hanford, Dewi Yokelson, August Knox, and Barry Rountree.
2025. HPC Benchmarking: Repeat, Replicate, Reproduce.
ACM Conference on Reproducibility and Replicability, July 29-31, Vancouver, Canada.

License
-------
Benchpark is released under the Apache 2.0 w/ LLVM Exception license. For more details see
the `LICENSE `_ file.

LLNL-CODE-850629

Owner

Citation (CITATION.cff)

# If you are referencing Benchpark in a publication, please cite the paper
# described here.
#
# Here's the raw citation:
#
#   Olga Pearce, Alec Scott, Gregory Becker, Riyaz Haque, Nathan Hanford, Stephanie Brink,
#   Doug Jacobsen, Heidi Poxon, Jens Domke, and Todd Gamblin. 2023.
#   Towards Collaborative Continuous Benchmarking for HPC.
#   In Workshops of The International Conference on High Performance Computing,
#   Network, Storage, and Analysis (SC-W 2023), November 12–17, 2023, Denver, CO, USA.
#   ACM, New York, NY, USA, 9 pages.
#
# And here's the CITATION.cff format:

cff-version: 1.2.0
title: "Benchpark"
message: "If you are referencing Benchpark in a publication, please cite the paper below."
repository-code: "https://github.com/llnl/benchpark"
preferred-citation:
  type: conference-paper
  doi: 10.1145/3624062.3624135
  url: https://doi.org/10.1145/3624062.3624135
  authors:
    - family-names: "Pearce"
      given-names: "Olga"
    - family-names: "Scott"
      given-names: "Alec"
    - family-names: "Becker"
      given-names: "Gregory"
    - family-names: "Haque"
      given-names: "Riyaz"
    - family-names: "Hanford"
      given-names: "Nathan"
    - family-names: "Brink"
      given-names: "Stephanie"
    - family-names: "Jacobsen"
      given-names: "Doug"
    - family-names: "Poxon"
      given-names: "Heidi"
    - family-names: "Domke"
      given-names: "Jens"
    - family-names: "Gamblin"
      given-names: "Todd"
  title: "Towards Collaborative Continuous Benchmarking for HPC"
  conference:
    name: "Supercomputing 2023 (SC’23)"
    city: "Denver"
    region: "Colorado"
    country: "US"
    date-start: 2023-11-12
    date-end: 2023-11-17
  month: 11
  year: 2023
  notes: LLNL-CONF-853344
authors:
  - family-names: "Pearce"
    given-names: "Olga"
  - family-names: "Scott"
    given-names: "Alec"
  - family-names: "Becker"
    given-names: "Gregory"
  - family-names: "Haque"
    given-names: "Riyaz"
  - family-names: "Hanford"
    given-names: "Nathan"
  - family-names: "Brink"
    given-names: "Stephanie"
  - family-names: "Jacobsen"
    given-names: "Doug"
  - family-names: "Poxon"
    given-names: "Heidi"
  - family-names: "Domke"
    given-names: "Jens"
  - family-names: "Gamblin"
    given-names: "Todd"

Issues and Pull Requests

Last synced: 6 months ago