Unexpected long reward time in example script

[baihuajun24]

Hi, I am running your example script grpo_7B.sh.

Environment:

• Machine: 8×H20 GPUs
• Modified hyperparameters: Changed max_response_length from 32k to 16k (other unchanged)
• Training steps: 200

Issue:
I ran training for 200 steps and observed the following timing distribution:

The figure plots timing_s/gen, timing_s/reward over timing_s/step per step. The reward time (orange line) dominates the step time.

Statistics:

• Mean distribution across all steps:
  • Rollout: ~30%
  • Reward: ~40%

Question:
I think the reward time is unexpectedly long. I don't know if this is expected behavior. Could you please help clarify?

[zzhbrr]

Hi, @baihuajun24 .
I have also noticed that the reward calculation takes too long. It is likely due to code generation tasks in the dataset, where the reward calculation requires executing code locally, and the execution process is serial, leading to very lengthy reward calculations. I do not yet have a good solution to this issue.

Additionally, I trained the model based on Qwen2.5-7B-Instruct and executed FastRL on an 8xH20 machine. However, I found that enabling speculative decoding resulted in negative gains. (I even observed that with a batch size of 4 and a draft token number of 4, the throughput with speculative decoding enabled was lower than when it was disabled.) I have not yet identified the reason. Could you share your experimental observations?

[Qinghao-Hu]

Hi @baihuajun24, thanks for reporting this issue!

As @zzhbrr explained, the longer reward calculation time is mainly caused by code execution during reward calculation. As discussed in our paper (Section 6.1), our experiments are conducted on a subset of Eurus, with a primary focus on math problems. We will release the sampled dataset used in our experiments to facilitate reproduction.

To mitigate this issue, asynchronous code execution can significantly reduce the reward computation overhead. However, our current implementation is based on an early version of VeRL, where this optimization is not yet supported. This limitation can be addressed once support for the latest VeRL AgentLoop is integrated.

[Qinghao-Hu]

Hi, @baihuajun24 . I have also noticed that the reward calculation takes too long. It is likely due to code generation tasks in the dataset, where the reward calculation requires executing code locally, and the execution process is serial, leading to very lengthy reward calculations. I do not yet have a good solution to this issue.

Additionally, I trained the model based on Qwen2.5-7B-Instruct and executed FastRL on an 8xH20 machine. However, I found that enabling speculative decoding resulted in negative gains. (I even observed that with a batch size of 4 and a draft token number of 4, the throughput with speculative decoding enabled was lower than when it was disabled.) I have not yet identified the reason. Could you share your experimental observations?

Hi @zzhbrr, as mentioned in our reply in #5, we use the Base model rather than the Instruct model. These two model types exhibit different response-length behaviors. And a significant drop in accepted length usually happens when switching target models.

Could you please try running the experiment with the Base model instead?