Multi-Task GRPO Training

[cbartmann]

I'm curious if anyone has figured out how to use GRPOTrainer for training a model on multiple objectives—like coding and math—simultaneously. Each objective would require its own reward function, which needs to be applied based on the task associated with each sample. It seems that batching samples by task might be necessary to fully leverage parallelism. Any insights or suggestions would be appreciated!

[shirinyamani]

Hi @cbartmann , thanks for the excellent point!
TRL now supports multi-task setup!
As an example, imagine you have a reward_func that checks if a code compiles, but the sample from the dataset isn't about code. Previously, in such a condition, the program would crash because the relevant reward_func would return None!
But NOT anymore!!!
This means now its OKAY to have mixed reward_funcs eventhough, some of them might not be used cuz there is no relevant sample for them!
This is the relevant PR that took care of it.

Thanks again and Happy Coding!