PyTupli

PyTupli is a Python library for creating, storing, and sharing benchmark problems and datasets for offline reinforcement learning (RL). PyTupli includes a lightweight client library with defined interfaces for uploading and retrieving benchmarks and data. It supports fine-grained filtering at both the episode and tuple level, allowing researchers to curate high-quality, task-specific datasets. A containerized server component enables production-ready deployment with authentication, access control, and automated certificate provisioning for secure use. By addressing key barriers in dataset infrastructure, PyTupli facilitates more collaborative, reproducible, and scalable offline RL research.

By using PyTupli, you can:

• ✅ Create benchmarks from any Gymnasium-compatible environment
• ✅ Share environments without exposing sensitive implementation details
• ✅ Record episode data from interactions with the environment or store static datasets associated to a benchmark
• ✅ Download datasets and convert them into formats compatible with popular offline RL libraries such as d3rlpy
• ✅ Store and manage artifacts like trained models or time series data

Installation

You can install PyTupli using pip:

bash pip install pytupli

Or if you're using Poetry:

bash poetry add pytupli

Development Setup

For local development in editable mode, clone the repository, navigate to the package directory and run bash poetry install

Optional Dependencies

PyTupli has several optional dependency groups that can be installed based on your needs:

Machine Learning Components: To install PyTorch for advanced quality metrics: bash poetry install --with ml Note: QFunctionMetric requires PyTorch. GeneralizedBehavioralEntropyMetric only requires PyTorch if you provide an observation_encoder. If you try to use these without installing PyTorch, you'll get a helpful error message.

Server Components: To install dependencies for running the PyTupli server: bash poetry install --with server

Documentation: To build the documentation: bash poetry install --with docs

Testing: To run tests: bash poetry install --with tests

You can combine multiple groups: bash poetry install --with server,docs,tests

Deployment

For deployment instructions, please refer to the deployment documentation.

Access Management

For a detailed guide of PyTupli's access management, please refer to the access management documentation.

CLI Usage

PyTupli provides a command-line interface for the TupliAPIClient. After deployment, first log in to the server and specify the URL:

bash pytupli login --username your_username --password your_password --url http://your-server:port

The URL will then be remembered for all future interactions with the server. As an alternative to handing it over during login, you can call bash pytupli set_url --url http://your-server:port

For user management, you can create new users and change passwords (requires admin privileges): ```bash

Create a new user

pytupli signup --username newuser --password initialpassword

Change a user's password

pytupli changepassword --username targetuser --password new_password ```

Other useful utilities of the CLI are listing benchmarks or artifacts: ```bash

List available benchmarks

pytupli list_benchmarks

List episodes for a benchmark

pytupli list_artifacts ```

You can get detailed help on any command by using the --help flag: ```bash

Show all available commands

pytupli --help

Get help on a specific command

pytupli command_name --help ```

Note: If you have IPython installed, it must be version <8.4 for the help functionality to work properly. This is due to a known issue that has been fixed in the underlying Fire library but is not yet available in the latest release.

Basic Usage Example

PyTupli makes it easy to create and share reinforcement learning benchmarks and associated datasets for offline RL. Here's an example of how a collaborative offline RL project based on PyTupli might look like:

1. Organization A has developed an environment for their specific use case (e.g., an energy management system). They have some historix data that they want to use to train an offline RL baseline.
2. They wrap their environment using PyTupli's wrapper classes to standardize the interface
3. They can then store and publish the benchmark through the PyTupli API.
4. They upload the historic data as episodes associated to the newly-created benchmark, making it available to other organizations.
5. Organization B can access the benchmark and download the data.
6. Before training their algorithms on it, B can filter the dataset, for example, for data created during a specific time period.
7. Finally, trained agents can be uploaded as artifacts associated to the benchmark

Code example (abbreviated):

```python

Organization A: Instantiate API storage object

tuplistorage = TupliAPIClient() tuplistorage.set_url("https://company-a-server.com/api")

Instantiate gymnasium environment

custom_env = PowerSystemEnv()

Wrap environment

tuplienv = TupliEnvWrapper(env=customenv, storage=tupli_storage)

Store and publish the benchmark

tuplienv.store( name='EMSbenchmark', description="Energy management system control task" ) tupli_env.publish()

Load the historical data

historicepisodes = loadhistoric_data()

Record and publish the episodes

for eps in historicepisodes: epsitem = Episode( benchmarkid=tuplienv.id, metadata=eps.metadata, tuples=eps.tuples ) epsheader = tuplistorage.record(epsitem) tuplistorage.publish(eps_header.id)

Organization B: Instantiate API storage object

tuplistorage = TupliAPIClient() tuplistorage.set_url("https://company-a-server.com/api")

We assume that this is the id of the previously stored benchmark

stored_id = "dl345kn456mlkl230"

Download benchmark

loadedtuplienv = TupliEnvWrapper.load( storage=tuplistorage, benchmarkid=stored_id )

Create dataset containing all episodes recorded during the summer months

months = ["June", "July", "August"] filtersummer = FilterOR(filters=[FilterEQ(key="month", value=m) for m in months]) filterbenchmark = FilterEQ(key='id', value=storedid) datasetsummer = TupliDataset( storage=tuplistorage ).withbenchmarkfilter(filterbenchmark).withepisodefilter(filtersummer) datasetsummer.load()

Convert to d3rlpy dataset

obs, act, rew, term, trunc = datasetsummer.convertto_tensors() ``` For a comprehensive guide covering most of PyTupli's functionality, including recording episodes, managing artifacts, and creating datasets, please refer to the Introduction.ipynb tutorial in the docs/source/tutorials directory.

Documentation

A detailed documention of the client library as well as our tutorials are available on ReadtheDocs.

Reference

PyTupli is maintained by the Cyber-Physical Systems Group at the Chair for Robotics and Embedded Systems at Technical University of Munich.

If you use PyTupli, please include the following reference @article{markgraf2025pytupli, title={PyTupli: A Scalable Infrastructure for Collaborative Offline Reinforcement Learning Projects}, author={Markgraf, Hannah and Eichelbeck, Michael and Cappey, Daria and Demirt{\"u}rk, Selin and Schattschneider, Yara and Althoff, Matthias}, journal={arXiv preprint arXiv:2505.16754}, year={2025} }