refactor: refactor env and data processor & add nemotron super 49b recipes

[yuki-97]

Follow up of #1472. Thanks @nv-mmanohara for adding this!

1. Add GRPO support for HelpSteer3 on LlamaNemotron 49B.
2. Add SFT support for tulu3 on LlamaNemotron 49B.
3. Add CodeJaccard environment.
4. Refactor env and data processor.
5. Introduce run_grpo.py, will [Refactor] Clear run_grpo_math.py and run_grpo_rm.py #1572 in a subsequent PR.

Test Result

grpo math before and after refactor

nemotron 49B

Known Issue

1. nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 cannot load from Hugging Face: [BUG] nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 cannot load from Hugging Face #1571
2. GRPO Nemotron HelpSteer3 recipe has very high logprob error: [BUG] GRPO Nemotron HelpSteer3 recipe has very high logprob error #1570

Design explaination

Purpose of task_name = data.task_name if hasattr(data, "task_name") else task_spec.task_name

(Answer to #1506 (review))
Relative doc is add to [docs/guides/grpo.md](docs/guides/grpo.md).

---

1.1 Enhanced Understandability

1. In the original run_grpo_math.py, the environment was hard-coded in the code. This file only supported one math environment, and the task_name of all datasets used was uniformly set to "math".
2. In this scenario, task_name, task_data_processors, and env were in a strict one-to-one binding. For example, the task_name of openmathinstruct2 was hard-coded as "math", the task_data_processors for the math task was bound to math_hf_data_processor, and the environment was bound to math_env.
3. Under this setup, one dataset could only be paired with one processor and one environment. We could interpret the task of run_grpo_math.py as "math", and the task of run_grpo_rm.py as "reward model".
4. Currently, we have abstracted run_grpo.py—the environment is no longer hard-coded but specified via configuration. This makes the binding between datasets, environments, and processors more flexible. For instance, openmathinstruct2 can use either the math environment or the reward model environment.
5. In this flexible setup, forcing task_name to "math" for all environments would cause confusion.
6. Our current design is dataset-centric: the dataset name serves as the task_name, and the task corresponding to the dataset can specify its own environment and processor.

1.2 Compatibility with Future Multi-Dataset and Multi-Environment Support

1. Consider a multi-dataset scenario where we use two datasets: openmathinstruct2 and dapo_math. Both are math-related datasets.
2. Suppose we want openmathinstruct2 (see: [openmathinstruct2.py#L38](

   RL/nemo_rl/data/datasets/response_datasets/openmathinstruct2.py

   Line 38 in 859a89a

   "task_name": "math",

   )) to use the math environment, and dapo_math (see: [dapo_math.py#L37](

   RL/nemo_rl/data/datasets/response_datasets/dapo_math.py

   Line 37 in 859a89a

   "task_name": "math",

   )) to use the reward model environment. We could theoretically specify the environment for each task in task_to_env (see: [run_grpo_math.py#L123](

   RL/examples/run_grpo_math.py

   Line 123 in 859a89a

   task_to_env["math"] = math_env

   )).
3. However, since the task_name for both datasets is hard-coded as "task_name": "math" in the code, this multi-environment configuration cannot be implemented.
4. But in current design, we can specify different task_name across datasets allowing them to use different env.

[github-actions]

⚠️ File Consistency Check

Check based on commit: cf6e02a (PR #1506 from yukih/pr-1472)

⚠️ Parallel Plans Synchronization Warning

The file nemo_rl/models/dtensor/parallelize.py was modified in this PR, but neither 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/optimized_tp_plans.py nor 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/parallelizer.py was updated.

Why this matters:
These files contain similar parallel plan implementations that should be kept synchronized to ensure consistency across the codebase.

Action required:

• Please review if the changes in nemo_rl/models/dtensor/parallelize.py should also be applied to 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/optimized_tp_plans.py or 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/parallelizer.py
• Update the appropriate related file(s) if necessary to maintain functional consistency
• Request access to the NVIDIA-NeMo/Automodel repository, create a PR against the nemo-rl-submodule branch, and update the Automodel submodule in the nemo-rl index
• Add @ffrujeri as a reviewer of this PR if you have any questions about the consistency requirements
• If the files are intentionally different, please add a comment in the PR explaining why

Files to check:

• Modified: nemo_rl/models/dtensor/parallelize.py
• Not modified: 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/optimized_tp_plans.py
• Not modified: 3rdparty/Automodel-workspace/Automodel/nemo_automodel/components/distributed/parallelizer.py

✅ DTensor Policy Worker Synchronization Check

Both DTensor policy worker files were modified in this PR:

• nemo_rl/models/policy/workers/dtensor_policy_worker.py
• nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py

Please ensure that the changes are consistent between both files where applicable.

---

_{This check ensures that related file implementations remain synchronized across the codebase. If you believe this warning is incorrect or the files should intentionally differ, please add a comment explaining the reasoning.}