https://github.com/amir22010/enas-pytorch

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"

https://github.com/amir22010/enas-pytorch

Repository

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"

https://github.com/Amir22010/ENAS-pytorch/blob/master/

# Efficient Neural Architecture Search (ENAS) in PyTorch

PyTorch implementation of [Efficient Neural Architecture Search via Parameters Sharing](https://arxiv.org/abs/1802.03268).



**ENAS** reduce the computational requirement (GPU-hours) of [Neural Architecture Search](https://arxiv.org/abs/1611.01578) (**NAS**) by 1000x via parameter sharing between models that are subgraphs within a large computational graph. SOTA on `Penn Treebank` language modeling.

**\*\*[Caveat] Use official code from the authors: [link](https://github.com/melodyguan/enas)\*\***


## Prerequisites

- Python 3.6+
- [PyTorch](http://pytorch.org/)
- tqdm, scipy, imageio, graphviz, tensorboardX

## Usage

Install prerequisites with:

    conda install graphviz
    pip install -r requirements.txt

To train **ENAS** to discover a recurrent cell for RNN:

    python main.py --network_type rnn --dataset ptb --controller_optim adam --controller_lr 0.00035 \
                   --shared_optim sgd --shared_lr 20.0 --entropy_coeff 0.0001

    python main.py --network_type rnn --dataset wikitext

To train **ENAS** to discover CNN architecture (in progress):

    python main.py --network_type cnn --dataset cifar --controller_optim momentum --controller_lr_cosine=True \
                   --controller_lr_max 0.05 --controller_lr_min 0.0001 --entropy_coeff 0.1

or you can use your own dataset by placing images like:

    data
     YOUR_TEXT_DATASET
        test.txt
        train.txt
        valid.txt
     YOUR_IMAGE_DATASET
        test
           xxx.jpg (name doesn't matter)
           yyy.jpg (name doesn't matter)
           ...
        train
           xxx.jpg
           ...
        valid
            xxx.jpg
            ...
     image.py
     text.py

To generate `gif` image of generated samples:

    python generate_gif.py --model_name=ptb_2018-02-15_11-20-02 --output=sample.gif

More configurations can be found [here](config.py).


## Results

Efficient Neural Architecture Search (**ENAS**) is composed of two sets of learnable parameters, controller LSTM ** and the shared parameters **. These two parameters are alternatively trained and only trained controller is used to derive novel architectures.

### 1. Discovering Recurrent Cells

![rnn](./assets/rnn.png)

Controller LSTM decide 1) what activation function to use and 2) which previous node to connect.

The RNN cell **ENAS** discovered for `Penn Treebank` and `WikiText-2` dataset:

 

Best discovered ENAS cell for `Penn Treebank` at epoch 27:



You can see the details of training (e.g. `reward`, `entropy`, `loss`) with:

    tensorboard --logdir=logs --port=6006


### 2. Discovering Convolutional Neural Networks

![cnn](./assets/cnn.png)

Controller LSTM samples 1) what computation operation to use and 2) which previous node to connect.

The CNN network **ENAS** discovered for `CIFAR-10` dataset:

(in progress)


### 3. Designing Convolutional Cells

(in progress)


## Reference

- [Neural Architecture Search with Reinforcement Learning](https://arxiv.org/abs/1611.01578)
- [Neural Optimizer Search with Reinforcement Learning](https://arxiv.org/abs/1709.07417)


## Author

Taehoon Kim / [@carpedm20](http://carpedm20.github.io/)

Owner

GitHub Events