Trainable Activations for Image Classification

We propose a set of the trainable activation functions — Cosinu-Sigmoidal Linear Unit (CosLU), DELU, Linear Combination (LinComb), Normalized Linear Combination (NormLinComb), Rectified Linear Unit N (ReLUN), Scaled Soft Sign (ScaledSoftSign), Shifted Rectified Linear Unit (ShiLU).

Pretrained weights.

CosLU

$$CosLU(x) = (x + \alpha \cos(\beta x))\sigma(x)$$

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

DELU

$$ DELU(x) = \begin{cases} SiLU(x), x \leqslant 0 \ (n + 0.5)x + |e^{-x} - 1|, x > 0 \end{cases} $$

$$SiLU(x) = x\sigma(x)$$

LinComb

$$LinComb(x) = \sum\limits{i=0}^{n} wi \mathcal{F}_i(x)$$

NormLinComb

$$NormLinComb(x) = \frac{\sum\limits{i=0}^{n} wi \mathcal{F}_i(x)}{\mid \mid W \mid \mid}$$

ReLUN

$$ReLUN(x) = min(max(0, x), n)$$

ScaledSoftSign

$$ScaledSoftSign(x) = \frac{\alpha x}{\beta + |x|}$$

ShiLU

$$ShiLU(x) = \alpha ReLU(x) + \beta$$

$$ReLU(x) = max(0, x)$$

INSTALLATION

Create venv. bash python3 -m venv venv

Activate venv. bash source venv/bin/activate

Install dependencies. bash pip install -r requirements.txt

PROJECT STRUCTURE

There are 3 main files - train.py, test.py, plot.py. You should run train.py first, then test.py, then plot.py.

Use whatever configuration you want to test. Configurations can be found in the configs folder, train.py and test.py use the same config. There are several plot configurations in the configs/plot folder.

There are many predefined run scripts in the scripts folder, just run one of them as .sh, scripts/train.sh and scripts/test.sh are scripts to train and test all possible configurations, scripts/plot.sh to plot results after training and testing.

All the results of the train / test phases are in the logs folder.

All proposed trainable activations are in activation.py.

HOW TO RUN

Let's say I want to train and test the ResNet-8 model with CosLU trainable activation on the CIFAR-10 dataset. python python train.py --config configs/coslu/cifar10/resnet8.yaml python test.py --config configs/coslu/cifar10/resnet8.yaml

If you want to train and test all proposed trainable activations with a specific model and dataset, you can use the script from the scripts folder. For example, train and test the DNN2 model on the MNIST dataset. bash sh scripts/dnn2_mnist.sh

Train and test all possible configurations. bash sh scripts/train.sh sh scripts/test.sh

Plot graphics for all configurations, it will work even if some configurations haven't been trained. bash sh scripts/plot.sh

CITATION

Project CITATION.

LICENSE

Project is distributed under MIT License.