Change services to actors except Policy and drop apps/rl

[DNXie]

Within all of our applications, everything is currently a Service. For example, in apps/grpo/main, trainer and replay_buffer should be actors.

Summary:

• Actors: dataloader, trainer, replay_buffer, compute_advantages
• Services: Policy, ref_model, reward_actor

Changes:

• Updated apps/grpo, apps/toy_rl (with dcp off)
• Updated Policy to take use_dcp from config.
• Dropped apps/rl since it is deprecated.
• Single actor call call_one instead of choose. The difference is that call_one makes sure the caller is a singleton.

Test

python -m apps.grpo.main --config apps/grpo/qwen3_8b.yaml
python -m apps.grpo.main --config apps/grpo/qwen3_1_7b.yaml
python -m apps.toy_rl.sumdigits --config apps/toy_rl/sumdigits.yaml

I didn't run

python -m apps.grpo.main --config apps/grpo/qwen3_multinode.yaml

since the config is outdated. Looks like this one is already deprecated.

cc @Ritesh1905

[casteryh]

Would love some clarification on service vs policy.
Especially why we have this distinction at all.
cc @allenwang28

[allenwang28]

Would love some clarification on service vs policy. Especially why we have this distinction at all.

I am going to assume you mean service vs actor! It's a good question. For context, the main reason we have services in the first place is to exactly handle load balancing and fault tolerance. These are things which Monarch doesn't give you out of the box, but gives you the capabilities to implement. We surely want this for something like vLLM, or if we want the ability to spin up and load balance across multiple execution environments in-band.

We initially had everything just be services for simplicity, but in reality not everything needs to be a service. For replay buffer, trainer, etc., you don't need the routing capabilities. Additionally, the fault tolerance story for those are not as well defined as that of the policy/environments/reference model.

Since the world will also learn about Monarch, I think the layering is clearer this way - Actors are the base capabilities from Monarch, Service is a distinct abstraction that's built on top of it.

[allenwang28]

actually I was wrong - I think we only have trainer and replay buffer be actors, the rest are ok to keep as services

[DNXie]

It makes sense. Why are dataset, compute_advantages, and reward_actor also services? Do they need replicas?

[DNXie]

If num_replica=1, what's the difference between actor and service?

[DNXie]

Done!

[allenwang28]

sorry Danning, Dataset is an actor, compute_advantages is an actor, reward_actor is a service

[allenwang28]

lgtm let's just use call_one() wherever appropriate

[allenwang28]

lgtm let's just use call_one() wherever appropriate

[DNXie]

Fixed the typo here. CC @casteryh

[casteryh]

thanks!