Open RL is code repository that contains minimalistic implementations of a wide collection of reinforcement learning algorithms. The purpose of this repo is to make RL more approachable and easier to learn. As such, code in this repo is optimized for readability and consistency between algorithms.

Compared to machine learning, RL is still rather niche. As such, finding resources for learning RL is a bit more difficult. While implementations broadly exist for two algorithms, Q-networks and vanilla policy gradients, it's much more difficult to find easy-to-follow implementations of others. For many of the algorithms implemented here, no simple implementations appear to exist whatsoever. Interestingly, it's not just state-of-the-art algorithms that haven't been re-implemented in an easy-to-follow way. It's also hard to find clear implementations of foundational algorithms like multi-armed bandits. It's for these reasons why open-rl was created! Happy learning!

Algorithms

In this repo you will find implementations for the following algorithms.

Model free learning

Policy-based methods

| | Discrete | Continuous | | --- | :---: | :---: | | REINFORCE | :heavycheckmark: | :heavymultiplicationx: | | REINFORCE w/ baseline | :heavycheckmark: | :heavymultiplicationx: | | VPG | :heavycheckmark: | :heavycheckmark: |

Value-based methods

| | Discrete | Continuous | | --- | :---: | :---: | | DQN | :heavycheckmark: | :heavymultiplicationx: | | Double DQN | :heavycheckmark: | :heavymultiplicationx: | | Dueling DQN | :heavycheckmark: | :heavymultiplicationx: | | DRQN (for POMDPs) | :heavycheckmark: | :heavymultiplicationx: |

Actor-critic methods

| | Discrete | Continuous | | --- | :---: | :---: | | A2C | :heavycheckmark: | :heavymultiplicationx: | | A3C | :heavycheckmark: | :heavymultiplicationx: | | DDPG | :heavymultiplicationx: | :heavycheckmark: | | TD3 | :heavymultiplicationx: | :heavycheckmark: | | SAC | :heavymultiplicationx: | :heavycheckmark: | | PPO | :heavymultiplicationx: | :heavycheckmark: |

Bandits

Multi-armed bandits

| | Discrete | Continuous | | --- | :---: | :---: | | Pure Exploration | :heavycheckmark: | :heavymultiplicationx: | | Epsilon Greedy | :heavycheckmark: | :heavymultiplicationx: | | Thompson Sampling - Bernoulli | :heavycheckmark: | :heavymultiplicationx: | | Thompson Sampling - Gaussian | :heavycheckmark: | :heavymultiplicationx: | | Upper Confidence Bounds (UCB) | :heavycheckmark: | :heavymultiplicationx: |

Contextual bandits

| | Discrete | Continuous | | --- | :---: | :---: | | Linear UCB | :heavycheckmark: | :heavymultiplicationx: | | Linear Thompson Sampling | :heavymultiplicationx: | :heavymultiplicationx: | | Neural-network approach | :heavycheckmark: | :heavymultiplicationx: |

Model-based learning

| | Discrete | Continuous | | --- | :---: | :---: | | Dyna-Q | :heavycheckmark: | :heavymultiplicationx: | | Deep Dyna-Q | :heavycheckmark: | :heavymultiplicationx: | | Monte-Carlo Tree Search (MCTS) | :heavycheckmark: | :heavymultiplicationx: | | MB + Model Predictive Control | :heavymultiplicationx: | :heavycheckmark: | | Model-Based Policy Opitmization (MBPO)| :heavymultiplicationx: | :heavycheckmark: |

Offline (batch) learning

| | Discrete | Continuous | | --- | :---: | :---: | | Conservative Q-learning (CQL) | :heavycheckmark: | :heavymultiplicationx: | | Model-Based Offline Reinforcement Learning (MOReL) | :heavycheckmark: | :heavymultiplicationx: | | Model-Based Offline Policy Optimization (MOPO) | :heavymultiplicationx: | :heavycheckmark: |

Other

| | Discrete | Continuous | | --- | :---: | :---: | | Behavioral Cloning | :heavycheckmark: | :heavymultiplicationx: | | Imitation Learning | :heavycheckmark: | :heavymultiplicationx: |

Installation

• Make sure you have Python 3.7 or higher installed
• Clone the repo ``` # Clone repo from github git clone --depth 1 https://github.com/natetsang/open-rl

Navigate to root folder

cd open-rl - Create a virtual environment (Windows 10). Showing instructions from `virtualenv` but there are other options too!

If not already installed, you might need to run this next line

pip install virtualenv

Create virtual environment called 'venv' in the root of the project

virtualenv venv

Activate environment

venv\Scripts\activate - Download requirements pip install -r requirements.txt ```

Contributing

If you're interested in contributing to open-rl, please fork the repo and make a pull request. Any support is much appreciated!

Citation

If you use this code, please cite it as follows: @misc{Open-RL, author = {Tsang, Nate}, title = {{Open-RL: Minimalistic implementations of reinforcment learning algorithms}}, url = {https://github.com/natetsang/open-rl}, year = {2021} }

Acknowledgements

This repo would not be possible without the following (tremendous) resources, which were relied upon heavily when learning RL. I highly recommend going through these to learn more. * CS285 @ UC Berkeley - taught by Sergey Levine * Grokking Deep RL book by @mimoralea * More to be added soon!