SUMO-RL

SUMO-RL provides a simple interface to instantiate Reinforcement Learning (RL) environments with SUMO for Traffic Signal Control.

Goals of this repository: - Provide a simple interface to work with Reinforcement Learning for Traffic Signal Control using SUMO - Compatibility with gymnasium.Env and popular RL libraries such as stable-baselines3 and RLlib - Easy customisation: state and reward definitions are easily modifiable

Reinforcement Learning based Smart Traffic Management System using RFID AND Image Detection

Changes from original repository ((https://github.com/LucasAlegre/sumo-rl): - To compare the efficacy of Q-learning/DQN/SARSA Reinforcement Learning agents on 2-way-single intersection - To simulate the best performing reinforcement learning algorithm on a 2-way single intersection using Image Detection and RFID-like monitoring systems

The main class is SumoEnvironment. If instantiated with parameter 'single-agent=True', it behaves like a regular Gymnasium Env. For multiagent environments, use env or parallel_env to instantiate a PettingZoo environment with AEC or Parallel API, respectively. TrafficSignal is responsible for retrieving information and actuating on traffic lights using TraCI API.

For more details, check the documentation online.

Install

Install SUMO latest version:

bash sudo add-apt-repository ppa:sumo/stable sudo apt-get update sudo apt-get install sumo sumo-tools sumo-doc Don't forget to set SUMOHOME variable (default sumo installation path is /usr/share/sumo) ```bash echo 'export SUMOHOME="/usr/share/sumo"' >> ~/.bashrc source ~/.bashrc Important: for a huge performance boost (~8x) with Libsumo, you can declare the variable: bash export LIBSUMOASTRACI=1 ``` Notice that you will not be able to run with sumo-gui or with multiple simulations in parallel if this is active (more details).

Install SUMO-RL, Stable Baselines 3, Gymnasium to run project

Stable release version is available through pip bash pip install sumo-rl stable_baselines3 gymnasium

Setup and activate virtual environment to run simulation:

```bash python -m venv env

On Windows

env\Scripts\activate

On Unix/MacOS

source env/bin/activate ``` <!-- end setup -->

MDP - Observations, Actions and Rewards

Observation

The default observation for each traffic signal agent is a vector: python obs = [phase_one_hot, min_green, lane_1_density,...,lane_n_density, lane_1_queue,...,lane_n_queue] - phase_one_hot is a one-hot encoded vector indicating the current active green phase - min_green is a binary variable indicating whether mingreen seconds have already passed in the current phase - ```laneidensityis the number of vehicles in incoming lane i dividided by the total capacity of the lane -lanei_queue```is the number of queued (speed below 0.1 m/s) vehicles in incoming lane i divided by the total capacity of the lane

You can define your own observation by implementing a class that inherits from ObservationFunction and passing it to the environment constructor.

Action

The action space is discrete. Every 'delta_time' seconds, each traffic signal agent can choose the next green phase configuration.

E.g.: In the 2-way single intersection there are |A| = 4 discrete actions, corresponding to the following green phase configurations:

Important: every time a phase change occurs, the next phase is preeceded by a yellow phase lasting yellow_time seconds.

Rewards

The default reward function is the change in cumulative vehicle delay:

That is, the reward is how much the total delay (sum of the waiting times of all approaching vehicles) changed in relation to the previous time-step.

You can choose a different reward function (see the ones implemented in TrafficSignal) with the parameter reward_fn in the SumoEnvironment constructor.

It is also possible to implement your own reward function:

```python def myrewardfn(trafficsignal): return trafficsignal.getaveragespeed()

env = SumoEnvironment(..., rewardfn=myreward_fn) ```

RESCO Benchmarks

In the folder nets/RESCO you can find the network and route files from RESCO (Reinforcement Learning Benchmarks for Traffic Signal Control), which was built on top of SUMO-RL. See their paper for results.

Experiments

Check experiments for examples on how to instantiate an environment and train your RL agent.

Q-learning algorithm in a two-way intersection:

bash python experiments/ql_single-intersection.py

stable-baselines3 DQN algorithm in a 2-way single intersection:

Obs: you need to install stable-baselines3 with pip install "stable_baselines3[extra]>=2.0.0a9" for Gymnasium compatibility. bash python experiments/dqn_2way-single-intersection.py

SARSA algorithm in a two-way single intersection:

bash python experiments/sarsa_2way-single-intersection.py

[Q-learning algorithm + simulation with dummy RFID and Image Detection monitoring] in a two-way single intersection:

bash python experiments/ql_rfid_monitoring.py -gui -s 100000