[Feature] OpenSpiel integration/support

[lanctot]

There has been some initial discussion regarding integration between ACME and OpenSpiel to enable large-scale RL in games. I'm tagging everyone who has been involved, hopefully we can use this thread to coordinate: @jhtschultz @fastturtle @mwhoffman @bshahr.

My first questions usually concern the use cases for games, things like legal versus illegal actions (more generally, a subset of actions being available at state s, A(s) \in A, from a larger "full set" A), and handling of things multiple agents receiving observations and submitting actions, maybe using turn-based dynamics over states etc.

Regarding, legal actions, since OpenSpiel includes the legal actions in its observations, at first glance one solution seems like changing the policy network (e.g. here: https://github.com/deepmind/acme/blob/9dd7058e1ca3b21721f0f10ad0d8a15fbbcb1cee/acme/agents/tf/mpo/agent.py#L49) to one that knows about illegal moves actions and handles them accordingly. One thing has been problematic in the past: it's over the algorithm itself that needs to know about the legal actions, for example DQN when it computes its target value, it needs to apply its max_{a' \in A(s')) Q(s',a') which requires the legal actions. @fastturtle this seems like it might require customizing the individual agents. Do you agree? If so, do you see an easy way out of hand-customizing all the agents?

Regarding the agent interactions with the environment, it seems like we could write a custom Environment Loop that would call into the underlying OpenSpiel state and apply actions as necessary, distribute rewards and observations. This seems similar to what is currently done in the OpenSpiel rl_environment and example learning loop.

I will start there. I would be delighted if we can achieve some smooth integration across these projects!

[fastturtle]

One thing has been problematic in the past: it's over the algorithm itself that needs to know about the legal actions, for example DQN when it computes its target value, it needs to apply its max_{a' \in A(s')) Q(s',a') which requires the legal actions.

This is a good point. It seems like this wouldn't apply to actor-critic algorithms, right? For DQN we could emit -inf for invalid actions, would that be enough for your example? I realize that handling this on a case-by-case basis (versus making the agent aware of valid actions) would push some complexity into your networks, but hopefully not too much. What do you think?

[jhtschultz]

My apologies for not jumping in here sooner. This is an exciting project and I believe integrating OpenSpiel and Acme will unlock a ton of potential. Acme has a wonderful (and growing!) collection of RL algorithms, and OpenSpiel surely has the largest collection of games with a standardized API designed specifically with RL in mind. It only stands to reason they should work together.

I currently have an interface that runs Acme’s DQN on OpenSpiel games. It successfully converges to an optimal policy in tic tac toe.

This necessitated a number of modifications. OpenSpiel’s rl_environment is not a dm_env, so it needed a wrapper. The environment loop had to be customized to support multiple agents. This added a few complications, but my approach was to keep the loop as simple as possible and create a specialized OpenSpiel agent to handle the extra book keeping.

For instance, after an agent acts in a multiplayer environment, there may be multiple intervening time steps before that agent acts again. We have to store the last action taken, and only add states to an agent's experience tuples when that agent is the active player (assuming a standard DQN setup). I handle this by having the OpenSpielAgent keep track of its actions, observe every time step, and only process the time step if that agent is the active player.

Handling illegal actions requires slight modifications to both the DQN learner and agent. I found that while trfl's epsilon_greedy() (used in the tensorflow DQN) supports legal actions masking, there was a bug in that functionality (see google-deepmind/trfl#28). We also need to modify the generic FeedForwardActor because select_action() must consider legal actions. Fortunately the reverb adders already had a slot extras that could be used for storing legal actions, so all of that code works as is.

I’m continuing to refine the interface and I believe it can be streamlined quite nicely. Everything OpenSpiel related can reside in a single dedicated directory. Perhaps I can open up a pull request here to get the ball rolling. Looking forward to any feedback the Acme team might have.

[mwhoffman]

This sounds great! We're excited to try and get this in and I think we can continue this conversation on the PR. At a high level I think we would want to avoid forking the DQN agent itself to allow for this type of integration, but modifications to the environment loop, wrappers, etc. are all fair game (and are exactly what I would have thought for this type of situation).