Benchmarks for AutoAlbument - AutoML for Image Augmentation.

Results

CIFAR-10 (Classification)

• Model: Wide-Resnet-28-10.
• Baseline augmentation strategy: Horizontal Flip with probability 0.5.
• Configs: AutoAlbument augmentation search | Baseline training | AutoAlbument training.

| Augmentation strategy | Top-1 Accuracy | Top-5 Accuracy | |---------------------------|:--------------:|:--------------:| | Baseline | 91.79 | 99.63 | | AutoAlbument | 96.02 | 99.91 |

SVHN (Classification)

• Model: Wide-Resnet-28-10.
• Both train and extra sets are used for training.
• Baseline augmentation strategy: no augmentations.
• Configs: AutoAlbument augmentation search | Baseline training | AutoAlbument training.

| Augmentation strategy | Top-1 Accuracy | Top-5 Accuracy | |---------------------------|:--------------:|:--------------:| | Baseline | 98.31 | 99.68 | | AutoAlbument | 98.48 | 99.72 |

ImageNet (Classification)

• Model: ResNet-50.
• Baseline augmentation strategy:
  • Resize an image to 256x256 pixels.
  • Crop a random 224x224 pixels patch.
  • Apply Horizontal Flip with probability 0.5.
• AutoAlbument augmentation strategy:
  • Resize an image to 256x256 pixels.
  • Crop a random 224x224 pixels patch.
  • Apply AutoAlbument augmentation policies.
• Configs: AutoAlbument augmentation search | Baseline training | AutoAlbument training.

| Augmentation strategy | Top-1 Accuracy | Top-5 Accuracy | |---------------------------|:--------------:|:--------------:| | Baseline | 73.27 | 91.64 | | AutoAlbument | 75.17 | 92.57 |

Pascal VOC (Semantic segmentation)

• Model: DeepLab-v3-plus.
• Baseline augmentation strategy:
  • Resize an image preserving its aspect ratio, so the longest size is 256 pixels.
  • If required, pad an image to the size 256x256 pixels.
  • Apply Horizontal Flip with probability 0.5.
• AutoAlbument augmentation strategy:
  • Resize an image preserving its aspect ratio, so the longest size is 256 pixels.
  • If required, pad an image to the size 256x256 pixels.
  • Apply AutoAlbument augmentation policies.
• Configs: AutoAlbument augmentation search | Baseline training | AutoAlbument training.

| Augmentation strategy | mIOU | |---------------------------|:--------------:| | Baseline | 73.34 | | AutoAlbument | 75.55 |

Cityscapes

• Model: DeepLab-v3-plus.
• Baseline augmentation strategy:
  • Resize an image preserving its aspect ratio, so the longest size is 256 pixels.
  • If required, pad an image to the size 256x256 pixels.
  • Apply Horizontal Flip with probability 0.5.
• AutoAlbument augmentation strategy:
  • Resize an image preserving its aspect ratio, so the longest size is 256 pixels.
  • If required, pad an image to the size 256x256 pixels.
  • Apply AutoAlbument augmentation policies.
• Configs: AutoAlbument augmentation search | Baseline training | AutoAlbument training.

| Augmentation strategy | mIOU | |---------------------------|:--------------:| | Baseline | 79.47 | | AutoAlbument | 79.92 |

How to run the benchmarks

1. Download datasets and put them in the following directory structure:
2. Clone this repository.
3. Run the run.sh script that will build a Docker image and train models using the following command:

./run.sh </path/to/data/directory> </path/to/outputs/directory>

e.g. ./run.sh ~/data ~/outputs

where - </path/to/data/directory> is a path to a directory that contains datasets (e.g., a directory that contains folders imagenet, pascal_voc, etc) - </path/to/outputs/directory> is a path to a directory that should contain outputs from a training pipeline, such as a CSV log with metrics and a checkpoint with the best model.