RL agent implementation

[tidiane-camaret]

Hello, I just stumbled on this repo, this is a great project.
I'm also currently implementing an RL agent for this game. I'm curious, which input format do you think would be relevant as for states/actions ? I have a feeling that simple station coordinates would be inefficient for training.

Best

[yanfengliu]

Hey,

Thanks for reaching out! At its current state this game is only playable with manual control. Here is a demo. Implementing an interface for training AI agents is my next goal. Feel free to fork and try things out!

For state, I am thinking a few stacked frames of the raw game screen should be a good choice. It is a generic approach and it was proved to work on Atari games. The nice thing about Mini Metro is that the full game state is shown on screen. There is no hidden information, just like Chess and Go. The resolution of the game probably needs to be set lower than 1920 x 1080 though, which means that everything needs to be scaled as a percentage of width and height instead of hard-coded pixel lengths.

For action, if we think from a point-and-click perspective, there is going to be be an infinitely large action space or a continuous space, which is probably very hard to get working. Mini Metro doesn't really have that many actions to take. At any given time we only need to make a decision on:

1. Make new connections between a set of stations
2. Cancel one of the existing paths
3. Do nothing

In the future as I add in more features, the action types might expand to include:

4. Add carriage
5. Augment station
6. Add number of allowed paths

The main problem is the number of stations. In the original game, the stations appear gradually. For us there are two options:

1. Make the stations appear gradually but fix the number of maximum allowed stations in a game. Make the selections on nonexistent stations illegal as part of the action validation code.
2. Make all stations show up at once.