Advice on getting QMIX to work on overcooked

[abaisero]

Hi! I'm working a value function decomposition model, and I'm trying to get overcooked (v1) results for some baselines, including QMIX.

I understand that this is a library mainly to provide the environments, and possibly the development of the baselines is not your primary concern; but I have tried to take QMIX model from qmix_rnn.py, combine it with the CNN model for overcooked, and it just won't learn anything; I'm hoping maybe someone here has some insight?

notably the losses skyrocket on the early stages of the training. I've been trying many ways to stabilize the learning, but to little success. I tried mitigating this issue by reducing learning rates, decreasing polyak tau, increasing target update intervals, adding normalization layers and so on and so forth.

Is there anybody who has experienced similar issues learning with the Qmix model?

[abaisero]

I noticed one weird thing about the QMIX model; out of the 4 hypernetworks, one seems to be a single dense layer; is there a specific reason for this?

https://github.com/FLAIROx/JaxMARL/blob/b3fb1611e693ecfd69c34b54d76c3248880e1390/baselines/QLearning/qmix_rnn.py#L127C1-L145C18

[mttga]

hello, the hypernetwork implementation should be fine. We have seen instabilities with QMix elsewhere, but I don't think as severe. Maybe the reason could rely on the new network you're using. How are you combining the CNN with the RNN? Do the same type of network work with VDN/MAPPO?

[abaisero]

Overcooked is fully observable, so I'm not using RNNs; I'm just using the CNN as used by the VDN code, and adding a couple extra linear layers at most.

I have found some very mild improvements by simplifying the CNN, I believe; but still not great.

[alexunderch]

@mttga I was able to reproduce the problem: when using qmix (I left rnn as it is and added a cnn policy like for VDN) losses skyrocket.

Can share reports the code diff to look for any adjustments.