metriq-gym

metriq-gym is a Python framework for implementing and running standard quantum benchmarks on different quantum devices by different providers.

• Open – Open-source since its inception and fully developed in public.
• Transparent – All benchmark parameters are defined in a schema file and the benchmark code is reviewable by the community.
• Cross-platform – Supports running benchmarks on multiple quantum hardware providers (integration powered by qBraid-SDK)
• User-friendly – Provides a simple command-line interface for dispatching, monitoring, and polling benchmark jobs (you can go on with your life while your job waits in the queue).

Data generated by metriq-gym is intended to be published on https://metriq.info.

Join the conversation! - For code, repo, or theory questions, especially those requiring more detailed responses, submit a Discussion. - For casual or time sensitive questions, chat with us on Discord's #metriq channel.

Quick Start (Recommended)

Four easy steps to get started with metriq-gym!

1. Install metriq-gym directly in your Python environment using pip:

sh pip install metriq-gym

1. Download a benchmark configuration file from the schemas/examples/ directory

   sh curl -O https://raw.githubusercontent.com/unitaryfoundation/metriq-gym/refs/heads/main/metriq_gym/schemas/examples/wormhole.example.json

2. Dispatch it to a quantum device or simulator.

   sh mgym job dispatch wormhole.example.json -p local -d aer_simulator

3. Poll the job to get the results.

   sh mgym job poll latest

You will see the results of the benchmark printed in your terminal. E.g. sh {'device': 'aer_simulator', 'job_type': 'Wormhole', 'provider': 'local', 'results': {'expectation_value': 0.99658203125}, 'timestamp': '2025-07-29T20:31:17.978851', 'version': '0.1.3.dev0'}

Prerequisites

• Python (version 3.12 or newer)

Developer Setup

These instructions are for setting up a development environment if you plan to contribute to metriq-gym or run the latest version from source.

Prerequisites

Before you begin, ensure you have the following installed: * Python (version 3.12 or newer) * Poetry for managing dependencies

Cloning the Repository

First, clone the repository and navigate to the project directory:

sh git clone --recurse-submodules https://github.com/unitaryfoundation/metriq-gym.git cd metriq-gym

If you've already cloned the repo (with or without submodules), run:

bash git pull --recurse-submodules to keep it up to date.

Installation

Once you have poetry installed and the repository cloned, run:

sh poetry install from the root folder of the project, in order to install the project dependencies.

We recommend doing this in an isolated virtual environment. See Poetry documentation for more information on managing virtual environments.

If you use pyenv, here is a quick start guide to set up the environment and install all dependencies:

sh pyenv install 3.13 pyenv local 3.13 poetry install eval $(poetry env activate) All Python commands below should be run in the virtual environment.

Workflow

metriq-gym supports two type of resources: benchmarks and suites of benchmarks.

Single benchmarks

You can dispatch benchmark jobs by specifying a configuration file for the benchmark you wish to run.

sh mgym job dispatch <BENCHMARK_CONFIG> --provider <PROVIDER> --device <DEVICE>

Refer to the schemas/examples/ directory for example configuration files for supported benchmarks.

If running on quantum cloud hardware, the jobs will be added to a polling queue. The status of the queue can be checked with

sh mgym job poll <METRIQ_GYM_JOB_ID>

where <METRIQ_GYM_JOB_ID> is the assigned job ID of the job that was dispatched as provided by metriq-gym.

Alternatively, the poll action can be used without any argument to view all dispatched jobs, and select the one that is of interest.

sh mgym job poll

In order to export results to a JSON file, you can use the --json flag with the poll action. sh mgym job poll <METRIQ_GYM_JOB_ID> --json This will create a JSON file with the results and the metadata of the job identified by <METRIQ_GYM_JOB_ID>. By default, the JSON file will be saved in the current working directory with the name <METRIQ_GYM_JOB_ID>.json.

Using local simulators

For quick testing without access to cloud hardware, metriq-gym can dispatch jobs to a local simulator. At the moment the Qiskit Aer simulator is supported. Specify the local provider and the aer_simulator device. Example (from the project root directory):

sh mgym job dispatch metriq_gym/schemas/examples/qml_kernel.example.json --provider local --device aer_simulator

You can also create a noisy simulator based on an IBM backend by passing the backend name as the device:

sh mgym job dispatch metriq_gym/schemas/examples/qml_kernel.example.json --provider local --device ibm_<BACKEND>

Polling local simulator jobs works the same way:

sh mgym job poll <METRIQ_GYM_JOB_ID>

Suites of benchmarks

A suite is a collection of benchmarks that can be dispatched together. This is useful for running multiple benchmarks on the same device or provider. To dispatch a suite, create a JSON file suite.json that contains an array of benchmark configurations. For example:

json { "name": "test_suite", "description": "Just a test suite for the README", "benchmarks": [ { "name": "BSEQ", "config": { "benchmark_name": "BSEQ", "shots": 10 } }, { "name": "wormhole_7_qubits", "config": { "benchmark_name": "Wormhole", "num_qubits": 7, "shots": 1000 } }, } You can then dispatch the suite using the suite dispatch action:

sh mgym suite dispatch suite.json --provider <PROVIDER> --device <DEVICE> This will create a set of jobs for each benchmark in the suite and return a suite ID. You can poll the suite results using the suite poll action:

sh mgym suite poll <METRIQ_GYM_SUITE_ID>

Credential management

To run on hardware, each hardware provider offers API tokens that are required to interact with their quantum devices. In order to run on these devices, you will need to follow the instructions on the respective pages of the providers and obtain API keys from them.

The .env.example file illustrates how to specify the API keys once you have acquired them. You will need to create a .env file in the same directory as .env.example and populate the values of these variables accordingly.

View jobs

You can view all the jobs that have been dispatched by using the view action. This will display basic information about each job, including its ID, backend, job type, provider, and device.

sh mgym job view In order to view the details of a specific job (e.g., the parameters the job was launched with), you can use the view action and pass the job's id as argument, or select the job by index from the list of all dispatched jobs.

sh mgym job view <METRIQ_GYM_JOB_ID>

Example: Benchmarking Bell state effective qubits (BSEQ) on IBM hardware

The following example is for IBM, but the general workflow is applicable to any of the supported providers and benchmarks.

To run on IBM hardware, you will also require an IBM token. To obtain this, please visit the IBM Quantum Platform and include the API token in the local .env file as previously described.

The schemas/examples/ directory houses example JSON configuration files that define the benchmark to run. In this case, we use the bseq.example.json file as we want to run a BSEQ job. The following dispatches a job on the ibm-sherbrooke device for BSEQ.

sh mgym job dispatch metriq_gym/schemas/examples/bseq.example.json --provider ibm --device ibm_sherbrooke

We should see logging information in our terminal to indicate that the dispatch action is taking place:

sh INFO - Starting job dispatch... INFO - Dispatching BSEQ benchmark from metriq_gym/schemas/examples/bseq.example.json on ibm_sherbrooke... ... INFO - Job dispatched with ID: 93a06a18-41d8-475a-a030-339fbf3accb9

We can confirm that the job has indeed been dispatched and retrieve the associated metriq-gym job ID (along with other pieces of metadata).

sh +--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+ | Metriq-gym Job Id | Provider | Device | Type | Dispatch time (UTC) | +======================================+============+======================================================+================+============================+ | 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm | ibm_sherbrooke | BSEQ | 2025-03-05T10:21:18.333703 | +--------------------------------------+------------+------------------------------------------------------+----------------+----------------------------+

We can use the "poll" action to check the status of our job:

sh mgym job poll 93a06a18-41d8-475a-a030-339fbf3accb9

Doing so gives us the results of our job (if it has completed):

sh INFO - Polling job... BSEQResult(largest_connected_size=100, fraction_connected=0.7874015748031497)

In the event where the job has not completed, we would receive the following message instead

sh INFO - Polling job... INFO - Job is not yet completed. Current status: INFO - - d0wtyfhvx7bg008203b0: QUEUED (position 3) INFO - Please try again later.

As a convenience, while we could supply the metriq-gym job ID, we can also poll the job by running mgym job poll and then selecting the job to poll by index from our local metriq-gym jobs database.

sh Available jobs: +----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+ | | Metriq-gym Job Id | Provider | Device | Type | Dispatch time (UTC) | +====+======================================+============+======================================================+================+=============================+ | 0 | 93a06a18-41d8-475a-a030-339fbf3accb9 | ibm | ibm_sherbrooke | BSEQ | 2025-03-05T10:21:18.333703 | +----+--------------------------------------+------------+------------------------------------------------------+----------------+-----------------------------+ Select a job index:

Entering the index (in this case, 0), polls the same job.

sh Select a job index: 0 INFO - Polling job...

Contributing

First, follow the Developer Setup instructions above.

Style guide

We don't have a style guide per se, but we recommend that both linter and formatter are run before each commit. In order to guarantee that, please install the pre-commit hook with

sh poetry run pre-commit install immediately upon cloning the repository.

Tests

The entire suite of tests can be run with sh poetry run pytest

Unit tests only can be run with sh poetry run pytest -m "not e2e"

End-to-end tests only can be run with sh poetry run pytest -m e2e

Type checking

The project uses mypy for static type checking. To run mypy, use the following command: sh poetry run mypy

Documentation

The project uses Sphinx to generate documentation. To build the HTML documentation:

1. Navigate to the docs/ directory: sh cd docs/

Run the following command to build the HTML files: sh make html

Open the generated index.html file located in the _build/html/ directory to view the documentation.