CleanMARL

CleanMARL provides single-file, clean, and educational implementations of Deep Multi-Agent Reinforcement Learning (MARL) algorithms in PyTorch, following the same philosophy of CleanRL.

Main Features

• Implementations of key MARL algorithms: VDN, QMIX, COMA, MADDPG, FACMAC, IPPO, and MAPPO.

• A documentation for algorithms, code and training details.

• We support parallel environments and recurrent policies.

• Tensorboard and Weights & Biases logging.

We provide more details in our documentation.

You can find runs in Weights & Biases: Weights & Biases

Quick Start

Prerequisites:

• Python >=3.9

Installation:

git clone https://github.com/AmineAndam04/cleanmarl.git
cd cleanmarl
pip install .

To run experiment you can run for example:

python  cleanmarl/vdn.py --env_type="pz" --env_name="simple_spread_v3" --env_family="mpe" --use_wnb --wnb_project="cleanmarl-test" --wnb_entity="cleanmarl-test" --total_timesteps=1000000

python  cleanmarl/mappo.py --env_type="smaclite" --env_name="3m" 

To do next

• Save the checkpoints.
• Support continuous actions (MAPPO, IPPO, MADDPG, FACMAC).
• Support individual rewards.
• Support Jax-based environments (Jaxmarl ..).
• Fully Jitted training for Jax-based environments.

Algorithms Implemented

Algorithm	Variants Implemented
Value Decomposition Networks (VDN)	vdn.py vdn_lstm.py vdn_multienvs.py
QMIX	qmix.py qmix_lstm.py qmix_multienvs.py qmix_memefficient.py
Counterfactual Multi-Agent (COMA)	coma.py coma_lstm.py coma_multienvs.py coma_lstm_multienvs.py coma_lbf.py
Multi-Agent Deep Deterministic Policy Gradient (MADDPG)	maddpg.py maddpg_multienvs.py maddpg_lstm.py maddpg_lstm_multienvs.py
Factored Multi-Agent Centralized Policy Gradients (FACMAC)	facmac.py facmac_multienvs.py
Independent Proximal Policy Optimization (IPPO)	ippo.py ippo_lstm.py ippo_multienvs.py ippo_lstm_multienvs.py
Multi-Agent Proximal Policy Optimization (MAPPO)	mappo.py mappo_lstm.py mappo_multienvs.py mappo_lstm_multienvs.py