MMFN: Multi-Modal Fusion Net for End-to-End Autonomous Driving

official branch for IROS 2022 paper codes including collecting data, all benchmark in paper, training scripts and evaluations etc.

Quickly view

Background: How to efficiently use high-level information or sensor data in end-to-end driving. The whole architecture in MMFN from origin paper:

Scripts quick view in run_steps folder:

• phase0_run_eval.py : collect data/ run eval in select routes files or town map
• phase1_preprocess.py : pre-process data before training to speed up the whole training time
• phase2_train.py: after having training data, run this one to have model parameters files. pls remember to check the config (You can try training process on Docker also)

This repo also provide experts with multi-road consideration. Refer to more experts, pls check latest carla-expert repo

0. Setup

Install anaconda Shell wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh bash Anaconda3-2020.11-Linux-x86_64.sh source ~/.profile

Clone the repo and build the environment

Shell git clone https://github.com/Kin-Zhang/mmfn cd mmfn conda create -n mmfn python=3.7 conda activate mmfn pip3 install -r requirements.txt

For people who don't have CARLA [在内地的同学可以打开scripts换一下函数 走镜像下载更快点.. ]

```Shell chmod +x scripts/* ./scripts/setup_carla.sh

input version

10.1

auto download now ...

```

Download the vectorized lib with .so, will install based on your system version, for commercial reason, we cannot open this part of codes(C++) fully but get the lib file, with lib setting, we can still directly input the opendrive file and output the vectorized lane, pls see more usage in the code Shell ./scripts/setup_lib.sh

bashrc or zshrc setting: ```bash

<< Leaderboard setting

===> pls remeber to change this one

export CODEFOLDER=/home/kin/mmfn export CARLAROOT=/home/kin/CARLA_0.9.10.1

===> pls remeber to change this one

export SCENARIORUNNERROOT=${CODEFOLDER}/scenariorunner export LEADERBOARDROOT=${CODEFOLDER}/leaderboard export PYTHONPATH="${CARLAROOT}/PythonAPI/carla/":"${SCENARIORUNNERROOT}":"${LEADERBOARDROOT}":"${CARLAROOT}/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x8664.egg":"${CODEFOLDER}/teamcode":${PYTHONPATH} ```

1. Dataset

The data is generated with leaderboard/expert_agent/mmfn_pilot.py in 8 CARLA towns using the routes and scenarios files provided at leaderboard/data on CARLA 0.9.10.1

The dataset is structured as follows:

- TownX_{tiny,short,long}: corresponding to different towns and routes files - TownX_X_07_01_23_26_34: contains data for an individual route - rgb_front: multi-view camera images at 400x300 resolution - lidar: 3d point cloud in .npy format - measurements: contains ego-agent's position, velocity and other metadata - radar: radar point in .npy format - maps: birdview maps image - opendrive: opendrive xodr file

How to Generate

First, please modify the config files and on .zshrc or .bashrc remember to export your CARLA_ROOT as above said, Here is config file preview: ```bash

Seed used by the TrafficManager (default: 0)

port: 2000 trafficManagerSeed: 0

============== for all route test===============

debug: False

only for debug ===> or just test the agent

routes: 'leaderboard/data/onlyonetown.xml'

towns: ['Town01', 'Town02', 'Town06', 'Town07']

routes: 'leaderboard/data/training_routes/'

scenarios: 'leaderboard/data/alltownstraffic_scenarios.json'

====================== Agent =========================

track: 'MAP' # 'SENSORS' agent: 'teamcode/expertagent/autopilot.py' defaults: - agentconfig: expert ```

Please write great port according to the CARLA server, and inside the script it will try to use Epic or vulkan mode since opengl mode will have black point on raining day, check the folder path and carla root to your path, then directly run this: [Hine: If the GPU memeory is bigger than 12G, and CPU memory with big swap space will be better.]

```bash conda activate mmfn

<< Leaderboard setting

===> pls remeber to change this one

export CODEFOLDER=/home/kin/mmfn export CARLAROOT=/home/kin/CARLA_0.9.10.1

===> pls remeber to change this one

export SCENARIORUNNERROOT=${CODEFOLDER}/scenariorunner export LEADERBOARDROOT=${CODEFOLDER}/leaderboard export PYTHONPATH="${CARLAROOT}/PythonAPI/carla/":"${SCENARIORUNNERROOT}":"${LEADERBOARDROOT}":"${CARLAROOT}/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x8664.egg":"${CODEFOLDER}/teamcode":${PYTHONPATH}

python runsteps/phase0runeval.py port=2010 towns="['Town02', 'Town04', 'Town05', 'Town10']" resume=True ifopen_carla=True

if you have big GPU memory, you can start with these two simultaneously.

python runsteps/phase0runeval.py port=2000 towns="['Town01', 'Town03', 'Town06', 'Town07']" resume=True ifopen_carla=True ```

The dataset folder tree will like these one:

bash data └── train ├── Town06_tiny ├── Town06_00_07_01_23_40_19 ├── lidar ├── maps ├── measurements ├── opendrive ├── rgb_front └── radar ├── Town06_00_07_01_23_40_19 └── Town06_00_07_01_23_40_19

The scripts will start a CARLA with specific port number, example running like this:

Note: since official stable leaderboard have memory leak issue, check out here. Please check your memory and kill it to resume. Other notes for CARLA leaderboard check out here

2. Training

No need CARLA in these phase, please remember to modify the train.yml config file and especially ==modify the DATA PATH==

The record and visualization on training params use the wandb, please login before train, more details can be found at wandb.ai, You can disable wandb from config file using disabled

1. To speed up the training since the data process is somehow time consuming. Save the finished data:

   bash python run_steps/phase1_preprocess_data.py

2. DDP if you want to use multi-GPU through DDP or multi computers: nproc_per_node as using GPU，nnodes as computer number

   bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 python -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 run_steps/phase2_train_net.py

3. Or one single GPU can just run:

bash python run_steps/phase2_train_net.py

Note for Docker: There is a Dockerfile ready for building training environment, but please remember to using -v link the datasets folder to container.

Benchmark

fork from transfuser codes, the benchmark training file and process can run as following command:

bash python team_code/benchmark/aim/train.py --device 'cuda:0' python team_code/benchmark/cilrs/train.py --device 'cuda:0' python team_code/benchmark/transfuser/train.py --device 'cuda:0'

3. Evaluate

This part is for evaluating to result or leaderboard, you can also download the modal file and try upload to leaderboard through leaderbaord branch. The whole process is the same way like generate dataset

1. Download or Train a model file saved to log/mmfn_img or log/mmfn_vec or log/mmfn_rad. Pretrained weight can download here, check issue note it's not the best weight or maybe better than papers. Try train on your own dataset for fair comparsion.

2. Check config/eval.yaml and inside e2e.yaml file spefice model weight path.

   ```bash routes: 'leaderboard/data/onlyonetown.xml'

   ====================== Agent =========================

   track: 'MAP' # 'SENSORS' agent: 'teamcode/e2eagent/mmfn_imgnet.py' defaults:

   • agent_config: e2e

   ======== e2e.yaml file for model path ============

   modelpath: 'log/mmfnimg' ```

3. Running eval python script and see result json file in result Folder

   bash python run_steps/phase0_run_eval.py --config-name=eval

4. The first one will take 1-2 mins because we are in traffic light. Be patience..

If all setting is correct, you will see eval like this one, pls remember to change route to only_one_town for debug.

And a demo video in CARLA main window view (speed up) the weight I used is mmfn_vec in dowload link:

https://github.com/Kin-Zhang/mmfn/assets/35365764/02be250c-1ab5-46a8-be99-0480df215e87

Cite Us

latex @inproceedings{zhang2022mmfn, title={MMFN: Multi-Modal-Fusion-Net for End-to-End Driving}, author={Zhang, Qingwen and Tang, Mingkai and Geng, Ruoyu and Chen, Feiyi and Xin, Ren and Wang, Lujia}, booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages={8638--8643}, year={2022}, organization={IEEE} }

Acknowledgements

This implementation is based on codes from several repositories. Thanks for these authors who kindly open-sourcing their work to the community. Please see our paper reference part to get more information on our reference

❤️: LBC, WorldOnRails, Transfuser, carla-brid-view, pylot, CARLA Leaderboard 1.0, Scenario Runner 1.0