Hi @lanctot and @alexunderch, want to give a quick update before I submit a PR (and hear your thoughts)
Straight answer: Changed a few hyperparameters to make the model converge — but these were not random tweaks. Each change targeted a specific issue observed during debugging:
(a) change reinitialize_advantage_networks=False
(b) increase policy_network_train_steps
(c) increase num_traversal

Before assuming it’s “just hyperparameters,” I ran a few small diagnostics to confirm the implementation itself is fine:

Advantage network: can overfit a single sample perfectly → infra works
Single info_state batches: network reaches the theoretical minimum MSE given regret variance → training is fine
Policy network: when trained separately, loss decreases smoothly → component also fine

So the glitch seems a mixture of: a) high variance sampled regrets given a tiny game (not a lot info_states) give hard time for advantage network (I noticed ~5k regrets from 1 same info_state and see their stds are [0.6781, 0.9935], per axis, ranging -3 to 3). Plus, b) Policy network needs more iterations to converge (currently by default trained 1 times).

I also ran a 40 simulations to test policy values: -0.058 (SD=0.029) close to the theoretical –0.056 (–1/18). I know variance is still noticeable, but the mean passes a t-test. (I set random seed so this is reproducible)

Let me know if above looks right. If so, I will submit my PR.

Btw, also notice Jax and tf2 versions not working. Not sure if they're expected to be maintained.if you think it’s worth fixing, I’m happy to take a look (or help on other higher-priority issues if any).

Thanks!

reproducible_experiment.html

open_spiel/python/examples/deep_cfr_pytorch.py fails to converge #1376

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions