Introduction

This repository is intended to maintain a list of common agreed-upon benchmark analysis tasks that can be used to exemplify, test, and compare different languages and approaches used for analysis. Also listed here are public data files available to run these benchmarks on and the repositories of actual implementations of these benchmarks.

Functionality benchmarks

1. Plot the E_T^miss of all events.
2. Plot the p_T of all jets.
3. Plot the p_T of jets with |η| < 1.
4. Plot the E_T^miss of events that have at least two jets with p_T > 40 GeV.
5. Plot the E_T^miss of events that have an opposite-charge muon pair with an invariant mass between 60 and 120 GeV.
6. For events with at least three jets, plot the p_T of the trijet four-momentum that has the invariant mass closest to 172.5 GeV in each event and plot the maximum b-tagging discriminant value among the jets in this trijet.
7. Plot the scalar sum in each event of the p_T of jets with p_T > 30 GeV that are not within 0.4 in ΔR of any light lepton with p_T > 10 GeV.
8. For events with at least three light leptons and a same-flavor opposite-charge light lepton pair, find such a pair that has the invariant mass closest to 91.2 GeV in each event and plot the transverse mass of the system consisting of the missing tranverse momentum and the highest-p_T light lepton not in this pair.

For the motivations behind these benchmarks, see motivation.md. For a technical reference of the terms used in the benchmarks, see reference.md.

Input data files

• Converted to NanoAOD from 2012 CMS open data:
  • root://eospublic.cern.ch//eos/root-eos/benchmark/Run2012B_SingleMu.root (16 GiB, 53 million events)

Language implementations

|Repository|Language|Description| |----------|--------|-----------| |opendata-benchmarks|RDataFrame|RDataFrame is a componenent of ROOT that provides a high-level interface for analyzing TTrees and other data formats. Each task is solved with a simpler syntax useful in interpreted ROOT macros as well as a fully compiled C++ syntax for best performance. | |nail|NAIL (Natual Analysis Implementation Language)|| |groot|Go|Part of the Go-HEP project, groot is a pure Go package that provides read/write access to ROOT files| |coffea|Python + Numpy|Coffea builds on numpy and awkward-array for columnar data analysis in Python| |bamboo|Python + RDataFrame|The bamboo analysis framework provides a high-level Python interface to RDataFrame (technically an embedded domain-specific language)| |queryosity| C++ | Queryosity is a (semi-)structured data analysis library with support for arbitrary data types. | |Rumble|JSONiq (an XQuery dialect for JSON data)|Most data in ROOT files can be exposed in the JSON data model and can thus be processed by JSONiq. This implementation is targeted to be run on Rumble, a JSONiq implementation on top of Spark, but could be run by any other JSONiq processor.| |BigQuery|BigQuery's dialect of SQL|SQL is arguably the most wide-spread language for querying structured data. Since SQL:1999, it supports arrays and structured types and is thus, in principle, suited for typical HEP analyses, though not many implementations support these features. BigQuery's dialect is based on SQL:2011, supports the mentioned features, and has a few additional language constructs that make queries more concise.| |PrestoDB|PrestoDB's dialect of SQL |Like BigQuery, Presto has some support for arrays and structured types; however, it only has limited support for nested queries and a more verbose syntax than BigQuery.| |Amazon Athena|Athena's dialect of SQL|Athena is a fully-managed Query-as-a-Service system based on PrestoDB with attractive scalability and pricing but a few more limitations than Presto (most importantly, no support for user-defined functions).| |SQL++ (AsterixDB)|SQL++|AsterixDB is a Big Data platform specialized for semi-structured data. Its query language is thus designed to deal with nested data intuitively.| |UnROOT.jl|Julia|Pure Julia implementation utilizing packages developed by JuliaHEP as a demonstration of ease of use, flexibility, and peak performance at the same time for end-user analysis.| |Snowflake|Snowflake's dialect of SQL|Snowflake is a fully-managed Query-as-a-Service system that boasts high performance and scalability as a pure in-cloud database. Moreover, Snowflake adds support for the powerful VARIANT data type, specifically designed to efficiently store and process semi-structured data.|

Adding new benchmarks, data, or implementations

• Additional benchmarks or public data files can be suggested as GitHub issues on this project to start a discussion within the HSF Data Analysis Working Group community.
• Suggested modifications to the layout of this repository are also welcome as new GitHub issues.
• If you would like to add a repository with a new implementation of the benchmarks, go ahead and submit a pull request with the proposed changes.