Delete DCP saved checkpoints used for weight sync

[allenwang28]

With #176 - model_state_dict is kept in the local directory and is left to the user to clean up manually.

This does two things:

1. Deletes all versions that are not this policy version
2. Adds model_state_dict to gitignore
3. Removes reference actor test for now. These were good tests for compute_log_probs - unfortunately that functionality is now in the apps/grpo/main, which tries to import vLLM. Getting that working on CPU CI is out of scope for this

Tested manually on single node and with a simple test

[joecummings]

How do we know the policy isn't currently reading from these weights?

[casteryh]

I think cleanup belongs to Policy since the Policy actor will know if an earlier version is still needed.
We can implement cleanup as an endpoint of Policy and we call it in the main loop to make sure everything is in sync.
(Later) once we figure out how to make the policy actors talk to each other after update_weights we can potentially move it to the inside of policy and do it automatically.

[allenwang28]

HMM yeah good point. I think this requires more consideration. I don't want to introduce some tracking at this moment, maybe a fragile heuristic we can go with is "don't delete the last 2" since we're not going off policy more than 1? Can follow up more with #194 wdyt

[casteryh]

HMM yeah good point. I think this requires more consideration. I don't want to introduce some tracking at this moment, maybe a fragile heuristic we can go with is "don't delete the last 2" since we're not going off policy more than 1? Can follow up more with #194 wdyt

Sounds good. add the heuristics now just to make sure nothing breaks and we can add proper evict logic later.

[allenwang28]

ok done!

[casteryh]

lgtm