How to create a cooperative multi agent environment?

[eerkaijun]

Hi, I am hoping to create a cooperative multi agent environment. I saw your guide here (https://github.com/praveen-palanisamy/macad-gym/wiki/creating-new-learning-environments.md) regarding creating new environments but how should I do it for a cooperative setting? I am hoping to use this environment for a centralised training and decentralised execution algorithm. Thanks!

[praveen-palanisamy]

Hi @eerkaijun ,
(A related discussion on creating an Adversarial multi-agent environment exists)
The following steps

1. The Wiki page you referenced provides the set of changes necessary for creating a new/custom learning environment or scenario.
2. Then, for the cooperative multi-agent environment you are interested in, you can re-use this urban driving through a signalized intersection scenario where 3 cars are crossing the intersection. To make the Actors (car1, car2 and car3) cooperate with each other to cross the intersection, you can begin by customizing the reward function that encourages cooperation among the Agents for example, by introducing a reward component that values other Agent's goal accomplishment. You could start by changing the following reward function spec for car2:
   

   macad-gym/src/macad_gym/envs/intersection/urban_signal_intersection_3c.py

   Line 57 in d0a2a75

   "reward_function": "corl2017",

   
   to use your own custom reward function that has the new bonus reward for cooperating with other Agents.

For your reference, the Reward functions are implemented in reward.py.

Let's convert this into an Idea discussion and I will be happy to help you if you need any further assistance in putting this together as a pull request.

[eerkaijun]

Thanks! I will work on this and create a pull request. I am thinking that to create a cooperative environment, we could use a team-average reward (ie each agents has their own reward function but the goal is to optimise the average reward). How would you suggest me to change the reward function in reward.py, since each agent has a private reward function and a shared reward function?

Another question: is it possible to make an episodic environment for example if I want a highway scenario for a specific distance only?

[eerkaijun]

@praveen-palanisamy also how do I use a different scenario (for eg highway) other than the intersection?

[praveen-palanisamy]

Yeah, that sounds good.
You can have local and global (shared) reward components by sending the shared reward to all the agents as part of the observation (simulating a communication channel that exist between the agents).

For episodic environments based on a distance threshold, you can add this distance-checking routine to the environment's step(...) method and terminate the episode by setting done=True when the distance-check condition is met.

For using a different scenario, you just have to change the value of the scenarios field in the config. An example for which line you would change is shown below:

macad-gym/src/macad_gym/envs/intersection/urban_signal_intersection_3c.py

Line 59 in 38884ac

"scenarios": "SUIC3_TOWN3_CAR2",

You can use any of the existing scenarios in macad_gym/carla/scenarios.py or add your own scenarios and use that.