Distributional Value Function Factorization (DFAC) Framework

This is the official repository that contain the source code for the DFAC paper:

• [ICML 2021] DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

If you have any question regarding the paper or code, ask by submitting an issue.

An extended version of the paper has been published in the Journal of Machine Learning Research (JMLR) 2023. Please refer to the dfac-extended repository for more information.

Gameplay Video Preview

Learned policy of DDN on Super Hard & Ultra Hard maps:

https://youtu.be/MLdqyyPcv9U

Installation

Install docker, nvidia-docker, and nvidia-container-runtime. You can refer to this document for installation instructions.

Execute the following commands in your Linux terminal to build the docker image:

```sh

Clone the repository

git clone https://github.com/j3soon/dfac.git cd dfac

Download StarCraft 2.4.10

wget http://blzdistsc2-a.akamaihd.net/Linux/SC2.4.10.zip

Extract the files to StarCraftII directory

unzip -P iagreetotheeula SC2.4.10.zip mv SC2.4.10.zip ..

Build docker image

docker build . --build-arg DOCKER_BASE=nvcr.io/nvidia/tensorflow:19.12-tf1-py3 -t j3soon/dfac:1.0 ```

Launch a docker container:

sh docker run --gpus all \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ --rm \ -it \ -v "$(pwd)"/pymarl:/root/pymarl \ -v "$(pwd)"/results:/results \ -e DISPLAY=$DISPLAY \ --device /dev/snd \ j3soon/dfac:1.0 /bin/bash

Run the following command in the docker container for quick testing:

sh cd /root/pymarl python3 src/main.py --config=ddn --env-config=sc2 with env_args.map_name=3m t_max=50000

After finish training, exit the container by exit, the container will be automatically deleted thanks to the --rm flag.

The results are stored in ./results.

We chose to release the code based on docker for better reproducibility and the ease of use. For installing directly or running the code in virtualenv or conda, you may want to refer to the Dockerfile. If you still have trouble setting up the environment, open an issue and describe your encountered issue.

Reproducing

The following is the list of commands used for the experiments in the paper:

```sh

3s5zvs3s6z

python3 src/main.py --config=iql --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=512 python3 src/main.py --config=vdn --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=128 python3 src/main.py --config=qmix --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=128 python3 src/main.py --config=diql --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=256 python3 src/main.py --config=ddn --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=512 python3 src/main.py --config=dmix --env-config=sc2 with envargs.mapname=3s5zvs3s6z rnnhiddendim=256

6hvs8z

python3 src/main.py --config=iql --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=128 python3 src/main.py --config=vdn --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=128 python3 src/main.py --config=qmix --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=256 python3 src/main.py --config=diql --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=512 python3 src/main.py --config=ddn --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=512 python3 src/main.py --config=dmix --env-config=sc2 with envargs.mapname=6hvs8z rnnhiddendim=256

MMM2

python3 src/main.py --config=iql --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=256 python3 src/main.py --config=vdn --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=64 python3 src/main.py --config=qmix --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=64 python3 src/main.py --config=diql --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=512 python3 src/main.py --config=ddn --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=512 python3 src/main.py --config=dmix --env-config=sc2 with envargs.mapname=MMM2 rnnhiddendim=256

27mvs30m

python3 src/main.py --config=iql --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=256 python3 src/main.py --config=vdn --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=64 python3 src/main.py --config=qmix --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=64 python3 src/main.py --config=diql --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=512 python3 src/main.py --config=ddn --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=128 python3 src/main.py --config=dmix --env-config=sc2 with envargs.mapname=27mvs30m rnnhiddendim=128

corridor

python3 src/main.py --config=iql --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=256 python3 src/main.py --config=vdn --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=128 python3 src/main.py --config=qmix --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=256 python3 src/main.py --config=diql --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=512 python3 src/main.py --config=ddn --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=128 python3 src/main.py --config=dmix --env-config=sc2 with envargs.mapname=corridor rnnhiddendim=64 ```

If you want to modify the algorithm, you can modify the files in ./pymarl directly, without rebuilding the docker image or restarting the docker container.

Compare Baseline code with DFAC code

The code of DFAC is organized with minimum changes based on oxwhirl/pymarl for readibility. You may want to compare the baselines with their DFAC variants with the following commands:

```sh

Configs

diff pymarl/src/config/algs/iql.yaml pymarl/src/config/algs/diql.yaml diff pymarl/src/config/algs/vdn.yaml pymarl/src/config/algs/ddn.yaml diff pymarl/src/config/algs/qmix.yaml pymarl/src/config/algs/dmix.yaml

Agent

diff pymarl/src/learners/qlearner.py pymarl/src/learners/iqnlearner.py diff pymarl/src/modules/agents/rnnagent.py pymarl/src/modules/agents/iqnrnn_agent.py

Mixer

diff pymarl/src/modules/mixers/vdn.py pymarl/src/modules/mixers/ddn.py diff pymarl/src/modules/mixers/qmix.py pymarl/src/modules/mixers/dmix.py ```

For comparing all modifications based on all used packages, refer to this comparison link of all modifications.

Developing new Algorithms

Updaing Packages

Since this repository is frozen in old commits for reproducibility, you may want to use the newest packages:

• oxwhirl/sacred
• oxwhirl/smac
• oxwhirl/pymarl

For common baselines, you may want to refer to the following package which collected a bunch of baselines:

• hijkzzz/pymarl2

Inspect the Training Progress

You can inspect the training progress in real-time by the following command:

sh tensorboard --logdir=./results

Citing DFAC

If you used the provided code or want to cite our work, please cite the DFAC paper.

BibTex format:

@InProceedings{sun21dfac, title = {{DFAC} Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning}, author = {Sun, Wei-Fang and Lee, Cheng-Kuang and Lee, Chun-Yi}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9945--9954}, year = {2021}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/sun21c/sun21c.pdf}, url = {http://proceedings.mlr.press/v139/sun21c.html}, }

You will also want to cite the SMAC paper for providing the benchmark used in the paper.

License

To maintain reproducibility, we freezed the following packages with the commit used in the paper. The licenses of these packages are listed below:

• oxwhirl/sacred (at commit 13f04ad) is released under the MIT License
• oxwhirl/smac (at commit 8d2c42b) is released under the MIT License
• oxwhirl/pymarl (at commit dd92936) is released under the Apache-2.0 License

Further changes based on the packages above are release under the Apache-2.0 License.