This package is intended for running and benchmarking optimization algorithms in Pytorch. It could be used for

• retrieving training curves for standard methods (SGD, Adam) on standard benchmark problems (e.g. training a ResNet on CIFAR10/100),
• testing and benchmarking new optimization algorithms

Getting started

Install via

python setup.py

or in order to install in developer mode via

python -m pip install --editable .

Results

Currently, the repo implements some standard image classification architectures (ResNet, VGG, ViT) and datasets (MNIST, CIFAR, Imagenet). It also contains a toy language model training setup (character-level Shakespeare dataset).

Below is an overview of experiments for which we provide the code that generated the results (i.e. the model, dataset preprocessing and training setup) as well as the actual scores at the end of each epoch. An overview and all links are given in the table below.

| ID | Model | Dataset | Results |
|-----|--------|----------|----------| | cifar100resnet110 | ResNet110 code | Cifar100 code | record |
| cifar10resnet20 | ResNet20 code | Cifar10 code | record |
| cifar10vgg16 | VGG16 code | Cifar10 code | record |
| cifar10vit | Small ViT code | Cifar10 code | record | | imagenet32resnet18 | ResNet18 code | Imagenet32 code | record |
| mnistmlp | MLP (3-layer, ReLU) code | MNIST code | record |

For each experiment, the exact config can also be found under configs/ where the files are named according to the ID (and possibly an integer suffix).

How to use

The main runner functions are run.py (or run.ipynb if you prefer notebooks). Any experiment needs a config file, see e.g. configs/test.json. In general, the name of the config file serves as experiment ID; it is used later for storing the output, plotting etc.

There are two ways of specifying a config for run.py.

1) dict-type configs

• Here, the config JSON is a dictionary where you can specify at each key a list or a single entry.
• The same is true for the hyperparameters of each entry in the opt key of the config file.
• A cartesian product of all lists entrys will be run (ie. potentially many single training runs in sequence).
• Multiple repetitions can be done using the key n_runs. This will use different seeds for shuffling the DataLoader.

2) list-type configs

• The config JSON is a list, where each entry is a config for a single training run.
• This format is intended only when you want to launch multiple runs in parallel. You should first create a dict-type config, and then use utilities for creating temporary list-type configs (see an example here).

You can run an experiment with run.py or with run.ipynb. A minimal example is:

python run.py -i 'test' Run python run.py --help in the terminal for detailed options.

Once you launch experiments, the output is stored as JSON in output if no other directory is specified.

Output structure

Every single run stores output as a dictionary in the following way:

{'config': configuration of the experiment 'history': list of dictionary (one per epoch), see key names below 'summary': useful information such as start time and end time }

For the entries in history, the following keys are important:

• learning_rate: the learning rate value in that epoch. This is different to the lr key in config['opt'] if learning rate schedule is used.
• train_loss: loss function value over training set
• val_loss: loss function value over validation set
• train_score: score function (eg accuracy) over training set
• val_score: score function (eg accuracy) over validation set

In stepback.utils.py you can find several helper functions for merging or filtering output files.