How can I train on a dynamically generated dataset (based on the model’s own outputs) using GRPO Trainer?

[xtitx0327]

I’m new to RL and deep learning, so my question might seem simple. I would greatly appreciate any advice!

I’m using LLM to solve a multi-step sequence generation task. At each step:

• The model receives (a) a system prompt and (b) the current sequence generated so far.
• The model must choose one of three actions:

1. Append something new to the sequence
2. Remove/modify something already in the current sequence
3. Terminate the generation process

I have implemented multiple reward functions to evaluate each action (append/remove/modify/terminate). Thus, the RL loop is:

1. Model generates an action.
2. Reward functions evaluate that action and assign a reward.
3. Model updates itself based on these rewards.

Conceptually, this fits the GRPO training loop. The problem is, my training data is not a fixed (“static”) dataset—instead, it’s generated on‐the‐fly from the model’s own past outputs. According to #3213 , the current GRPO Trainer does not support IterableDataset.

Question: What’s the recommended way to handle a dynamically generated dataset with GRPO Trainer? Is there a workaround, or do I need to implement a custom training loop? Thank you for any pointers!

[The-Inscrutable-X]

I have the same question.