How to Perform SFT After RL When the Model Outputs "Think" Tokens?

[MSS444]

In a reinforcement learning (RL) pipeline, we typically follow this sequence:

Supervised Fine-Tuning (SFT) with Chain-of-Thought (COT) data
Reinforcement Learning (RL)
SFT on non-reasoning data
RL again, and so on.
However, after RL, the model starts generating "think" tokens or internal reasoning steps. This raises several questions:

1. How do we perform SFT after RL when the model now outputs "think" tokens?
2. What loss function should be used to align the model correctly?
3. Does fine-tuning on non-reasoning data cause the model to forget COT patterns learned earlier?
4. If we do not have COT data for initial SFT but only non-COT data, how does this impact the model when followed by RL?

Looking for insights, research papers, or references that address these concerns. 🚀