Add torchstore to policy

[joecummings]

spawn_service(
            ServiceConfig(procs_per_replica=1, with_gpus=True, num_replicas=1),
            Policy,
            config=PolicyConfig(
                worker_params=WorkerConfig(model=model),
                sampling_params=SamplingOverrides(
                    num_samples=group_size, max_tokens=16
                ),
            ),
            store=await MultiProcessStore.create_store(),
        ),

[pbontrager]

I'm just not sure from this how store is getting passed in but otherwise looks good

[pbontrager]

Should there be a check that the new version exists on the store?

[joecummings]

I was just thinking that it would fail when trying to lookup the key and that would be obvious enough for now, but I could make it more explicit.

[pbontrager]

Does spawn services need to know to pass this in?

[joecummings]

Yes, it has to be passed in from the top level so you can use the same for Trainer.

[joecummings]

I'm just not sure from this how store is getting passed in but otherwise looks good

Yeah, the launch logic IMO is really confusing - took me awhile to figure out why store was not being passed into the policy worker. This is a consequence of the launch vs init params of the service. Worth a discussion later as to which is preferred. Having both right now is just confusing.