Question about the runtime of main experiments in the RLPR paper

[hcx-stu]

Thank you for open-sourcing the code for RLPR. I’ve been studying the implementation with great interest, and I really appreciate the clarity of your work.

According to the paper, the experiments were conducted using 32 NVIDIA A100 GPUs. I was wondering if you could kindly share the approximate training time (or total runtime) for the main experiments reported in the paper. This information would be extremely helpful for reproducing the results.

[hcx-stu]

Furthermore, I would like to ask at which training step the results reported in the main experiments were obtained.

[jibo27]

Thank you for your interest! Our final Qwen model underwent 4 days and 13 hours of training (from step 0 to step 320). The figure below shows the runtime (in seconds) across training steps. Periodic peaks occur due to evaluation every 40 steps. We selected the 320-step checkpoint as it achieved optimal performance.

[hcx-stu]

Thank you for your interest! Our final Qwen model underwent 4 days and 13 hours of training (from step 0 to step 320). The figure below shows the runtime (in seconds) across training steps. Periodic peaks occur due to evaluation every 40 steps. We selected the 320-step checkpoint as it achieved optimal performance.

Thank you very much for your reply. We reproduced the experiment using 48 NVIDIA H100 GPUs, but unfortunately, our results did not meet expectations. We ran the experiment for 320 steps and observed the best performance at step 200; however, there is still a substantial gap compared to the results reported in the paper. Could you kindly share any insights into what might be causing this discrepancy?

Step	MMLU-Pro	GPQA	TheoremQA	WebInst.	MATH-500	Minerva	AIME 24	General	All
200	53.15	35.98	56.09	74.84	76.70	54.23	15.21	55.02	52.32
320	52.75	34.22	53.48	73.82	74.80	50.83	13.75	53.57	50.52

[jibo27]

Hi, you can first check whether the evaluation was conducted using LLM-as-a-judge or a rule-based approach. If the evaluation was not performed using LLM-as-a-judge, the scores may appear lower than expected, as some correct responses might be mistakenly marked as incorrect.

You can review the model's actual outputs and their correctness judgments in the saved test logs: data/logs/test_generations_*.

For reference, here is the WandB report link containing the full metrics from our model training. In this report, the performance was measured using LLM-as-a-judge (specifically, a Qwen instruct model). The final results were further refined based on manual verification of the outputs.

[hcx-stu]

Hi, you can first check whether the evaluation was conducted using LLM-as-a-judge or a rule-based approach. If the evaluation was not performed using LLM-as-a-judge, the scores may appear lower than expected, as some correct responses might be mistakenly marked as incorrect.

You can review the model's actual outputs and their correctness judgments in the saved test logs: data/logs/test_generations_*.

For reference, here is the WandB report link containing the full metrics from our model training. In this report, the performance was measured using LLM-as-a-judge (specifically, a Qwen instruct model). The final results were further refined based on manual verification of the outputs.

Thank you for your reply. We employed LLM-as-a-judge (Qwen/Qwen2.5-72B-Instruct) for evaluation. Although our experimental results differ somewhat from those reported in the paper, they nonetheless further substantiate the effectiveness of RLPR. I look forward to seeing your work published in the near future.

[PangziZhang523]

@jibo27 Hello, have you experimented with longer caption-like general domain tasks?

[jibo27]

@jibo27 Hello, have you experimented with longer caption-like general domain tasks?

Hi, we didn't test much on tasks like these, but this is an interesting direction!

[hcx-stu]

@jibo27 Hello, I have a question regarding the RLVR baseline mentioned in the paper. Could you please clarify what training data was used for RLVR?

Thank you in advance for your time and support! 🙏

[jibo27]

@jibo27 Hello, I have a question regarding the RLVR baseline mentioned in the paper. Could you please clarify what training data was used for RLVR?

Thank you in advance for your time and support! 🙏

Hi, the training data is the same as our training data, i.e., RLPR-Train-Dataset.

[hcx-stu]

@jibo27 Hello, sorry to bother you.

I noticed that the RLPR paper mentions reproducing the VeriFree work. We also attempted to replicate VeriFree, but unfortunately, we encountered an issue where the model collapses after around 1000 steps and starts producing nonsensical outputs. We reached out to the authors of VeriFree, but unfortunately, we did not receive a response.

Since I saw that you have also worked on reproducing related work, I wanted to ask if you’ve encountered a similar issue? If possible, would you be willing to share the VeriFree model you trained? It would be greatly appreciated!

Thank you so much!

[zyandtom]

Thank you for your interest! Our final Qwen model underwent 4 days and 13 hours of training (from step 0 to step 320). The figure below shows the runtime (in seconds) across training steps. Periodic peaks occur due to evaluation every 40 steps. We selected the 320-step checkpoint as it achieved optimal performance.

Thank you very much for your reply. We reproduced the experiment using 48 NVIDIA H100 GPUs, but unfortunately, our results did not meet expectations. We ran the experiment for 320 steps and observed the best performance at step 200; however, there is still a substantial gap compared to the results reported in the paper. Could you kindly share any insights into what might be causing this discrepancy?

Step MMLU-Pro GPQA TheoremQA WebInst. MATH-500 Minerva AIME 24 General All
200 53.15 35.98 56.09 74.84 76.70 54.23 15.21 55.02 52.32
320 52.75 34.22 53.48 73.82 74.80 50.83 13.75 53.57 50.52

so although it not reach the metric in paper, is it better than RLVR or model-based reward?