TIREx Tracker

Automatic resource and metadata tracking for information retrieval experiments.

The TIREx tracker is a command line tool and API to automatically track resource usage, hardware specifications, and other metadata when running information retrieval experiments. It can be used either from the command line, from C/C++ code, in Python applications, or in Java/Kotlin applications.

Command Line Tool

Download a prebuilt binary for your hardware architecture and operating system from the latest release.

After downloading, run tirex-tracker --help to get a full description of all supported command line arguments.

In most cases, you would use the TIREx tracker CLI like this:

shell tirex-tracker "<command>"

This will measure all hardware metrics and metadata while executing the given shell command <command>.

C/C++ API

To use the TIREx tracker in your C/C++ projects, first include this repository to your CMake file like this:

```cmake include(FetchContent)

Use GIT_TAG to request the tag (or branch) you would like

FetchContentDeclare(tirextracker GITREPOSITORY https://github.com/tira-io/tirex-tracker.git GITTAG 0.0.11) FetchContentMakeAvailable(tirextracker) targetlinklibraries( tirextracker::tirextracker) ```

This will link the TIREx tracker C API to your binaries. Take a look at the examples to see how our C API can be used.

A minimal example would is shown below:

```c

include

int main() { // Configure measures to track. tirexMeasureConf conf[] = { {TIREXTIMEELAPSEDWALLCLOCKMS, TIREXAGGNO}, // Further measures... tirexNullMeasureConf // sentinel value }; sizet pollIntervalMs = 100; tirexTrackingHandle* handle; tirexStartTracking(conf, pollIntervalMs, &handle);

// Do something...

tirexResult* result;
tirexStopTracking(handle, &result);
// Analyze the results.
tirexResultFree(result);

} ```

The TIREx tracker will automatically track the specified metrics and metadata for everything that is run between tirexStartTracking and tirexStopTracking.

You can customize the measures to track by adjusting the conf array in the example above.

Python API

First, install the TIREx tracker Python package from PyPI:

shell pip install tirex-tracker

Now, you can track the hardware metrics and metadata of your Python code by using the context manager:

```python from tirex_tracker import tracking

with tracking() as results: # Do something...

print(results) ```

Alternatively, you can track the hardware metrics and metadata of a Python function by using the function decorator:

```python from tirex_tracker import tracked

@tracked def do_something(): # Do something...

do_something()

print(do_something.results) ```

If you cannot use either the context manager or the function decorator from above, you can manually start and stop the tracking:

```python from tirextracker import starttracking, stop_tracking

handle = starttracking() try: # Do something... finally: results = stoptracking(handle)

print(results) ```

Java/Kotlin/JVM API

The Java/Kotlin API can be installed via Gradle or Maven from the Maven Central Repository. After installing the package, you can use the TIREx tracker JVM API in your Java or Kotlin projects.

Gradle Dependency

For Gradle, add the following line to the dependencies block of your build.gradle (or build.gradle.kts) file:

gradle dependencies { implementation("io.tira:tirex-tracker:x.x.x") }

Maven Dependency

For Maven projects, add these lines to your pom.xml file:

xml <dependency> <groupId>io.tira</groupId> <artifactId>tirex-tracker</artifactId> <version>x.x.x</version> </dependency>

Replace the version placeholder (x.x.x) with the latest available version tag.

You can now use the TIREx tracker JVM API in your Java or Kotlin projects.

Java Usage

For pure Java projects, the easiest way is to use the track function and pass your code to be tracked as a lambda like this:

```java package io.tira.tirex.tracker.example;

import io.tira.tirex.tracker.*;

void main() { var result = Tracker.track(() -> { // Do something... });

System.out.println(result);

} ```

Alternatively, use the try-with-resources syntax like this:

```java Tracked tracked = new Tracked(); try (tracked) { // Do something... }

System.out.println(tracked.result); ```

Kotlin Usage

In Kotlin projects, the track function takes an inline block so that your code can be tracked like this:

```kotlin package io.tira.tirex.tracker.example

import io.tira.tirex.tracker.*

fun main() { val result = track { // Do something... }

println(result) } ```

Tracked Measures

ir_metadata Extension

The resources, hardware specifications, and metadata tracked by the TIREx tracker can easily be exported to an ir_metadata file. Not all of our extensive metadata fits well into the current ir_metadata specification version 0.1. We therefore extend the specification and propose ir_metadata 0.2-beta:

Contributing

We happily accept pull requests, feature requests, and bug reports to the TIREx tracker! To get started for contributing to the development, first clone this repository:

shell git clone https://github.com/tira-io/tirex-tracker.git cd tirex-tracker

The further steps will depend on which part of the TIREx tracker's API you work on: the C API, the Python API, or the JVM API.

C development

[!TIP] If you are not familiar with CMake, we highly recommend using the CMake Visual Studio Code extension.

We use the meta-build tool CMake to configure the project. Different build targets are used so that you can select what to build. Doxygen is used to generate the documentation for the C API. If you haven't installed CMake and/or Doxygen, please install it as described here for CMake and here for Doxygen

First, configure CMake enable desired build targets by running the following. Here, we use GCC but any C++20 compliant compiler also works.

shell cmake -S c/ -B c/build/ \ -D CMAKE_BUILD_TYPE=Release \ -D CMAKE_C_COMPILER=gcc-13 \ -D CMAKE_CXX_COMPILER=g++-13 \ -D BUILD_SHARED_LIBS=YES \ -D TIREX_TRACKER_BUILD_DOCS=YES \ -D TIREX_TRACKER_BUILD_DEB=YES \ -D TIREX_TRACKER_BUILD_EXAMPLES=YES

(Hint: If you do not want to generate the documentation and have not installed Doxygen, you can disable it by setting TIREX_TRACKER_BUILD_DOCS=NO.)

To build the library, run

shell cmake --build c/build/ --config Release --target tirex_tracker

Under Linux, this will compile the C API into a statically linked library at c/build/extensions/libtirex_tracker.so. The supported targets are:

| Target | Type | Description | |:-----------------------|:--------------|:----------------------------------------------------------------------------------------------| | tirex_tracker | library | A shared library containing tirex_tracker and all extensions. | | measure | executable | The measure command. | | 01_tracking | executable | Example 01: demonstrating basic tracking. | | 02_list_measures | executable | Example 02: demonstrating how to fetch meta information through the API. | | 04_ir_extension | executable | Example 04: demonstrating the IR extension (measureext_ir). | | tirex_tracker_docs | documentation | The library documentation (only available if Doxygen is installed). | | package | package | The Debian package contining the measure command. |

That means, to build the Debian package, run:

shell cmake --build c/build/ --config Release --target package

You will find the compiled Debian package file at c/build/tirex-tracker-*-Linux.deb (where * is the version).

CMake Options

The below CMake options are used to include/exclude certain features from the build, which can improve the build time.

| Option | Description | Default | |:-------------------------------|:--------------------------------------------------------------------------------------------------------|:-------:| | TIREX_TRACKER_ONLY_DOCS | Build only the documentation (disables tests and others). | OFF | | TIREX_TRACKER_BUILD_EXAMPLES | Build the examples. | OFF | | TIREX_TRACKER_BUILD_DEB | Build the Debian package. | OFF | | TIREX_TRACKER_BUILD_DOCS | Build the documentation. | OFF | | TIREX_TRACKER_BUILD_CMD_DEB | Build the Debian package for the measure command (requires TIREX_TRACKER_BUILD_EXAMPLES to be ON) | OFF |

Python development

When developing (parts of) the Python API, first create a virtual environment and then install all necessary developer tooling by running:

shell python3 -m venv venv/ source venv/bin/activate pip install python/[tests]

Once the Python developer tools are set up like that, you can check the Python code, static typing, and security, and run all tests with:

shell ruff check python/ # Code format and LINT mypy python/ # Static typing bandit -c python/pyproject.toml -r python/ # Security pytest python/ # Unit tests

JVM Development

The JVM wrapper API uses Gradle for build and test. Consistent builds are ensured by using a Gradle wrapper. To build the classes, run:

shell jvm/gradlew --project-dir jvm/ build

Tests and checks for Java and Kotlin usage can be run by:

shell jvm/gradlew --project-dir jvm/ check

Citation

If you find our work useful and reference or use it in a paper, please cite us.

bibtex @inproceedings{tirextracker2025, author = {Hagen, Tim and Fr{\"o}be, Maik and Merker, Jan Heinrich and Scells, Harrisen and Hagen, Matthias and Potthast, Martin}, booktitle = {48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2025)}, month = jul, publisher = {ACM}, title = {{TIREx Tracker: The Information Retrieval Experiment Tracker}}, year = {2025} }

You can also use the CITATION.cff to generate a citation in other formats.

License

The TIREx tracker code is licensed under the MIT License. If you use the TIREx tracker in your experiments, we would appreciate you citing our paper.

Abstract

The reproducibility and transparency of retrieval experiments heavily depends on properly provided information on the experimental setup and conditions. But as manually curating such experiment metadata can be tedious, error-prone, and inconsistent, metadata should be systematically collected in an automatic way—similar to the collection of Python and git-specific settings in the ir_metadata reference implementation. To enable a platform-independent automatic metadata collection following the ir_metadata specification, we introduce the TIREx tracker: a tool realized via a lightweight C binary, pre-compiled with all dependencies for all major platforms to track hardware configurations, usage of power/CPUs/RAM/GPUs, and experiment/system versions. The TIREx tracker seamlessly integrates into Python, Java, or C/C++ workflows and can be easily incorporated in run submissions of shared tasks, which we showcase for the TIRA/TIREx platform.