[GOLD][vLLM] On-policy generation decodes prompts with skip_special_tokens=True (chat template stripped) — intentional?

[siyan-zhao]

Hi TRL team, thanks for GOLD and the vLLM integration.

I’m trying to understand the intended behavior of GOLD’s on-policy generation with vLLM, which was originally identified by @simran135 and discussed in https://github.com/siyan-zhao/OPSD/issues/3.

In v0.29.0, this line says:

https://github.com/huggingface/trl/blob/v0.29.0/trl/experimental/gold/gold_trainer.py#L1680

"Decode prompts for vLLM (without special tokens - vLLM expects clean text)"

In _generate_on_policy_outputs_vllm, prompts are decoded with skip_special_tokens=True before being sent to vLLM:

prompts_text_for_vllm = self.processing_class.batch_decode(
    inputs["prompts"],
    skip_special_tokens=True,  # strips <|im_start|>, <|im_end|>, etc.
)

This strips chat-template special tokens (e.g., <|im_start|>, <|im_end|>, role markers) before vLLM receives the prompt. Then vLLM gets raw text and tokenizes it, rather than receiving the original templated prompt IDs.

For chat-template-driven models (e.g., Qwen3), this may change rollout behavior. In particular, if thinking mode depends on template-time controls (such as template kwargs), it may not be activated during on-policy vLLM generation.

Questions

1. Is using skip_special_tokens=True here intentional?
2. What is the reason for requiring “clean text” instead of preserving templated prompt IDs/text?

Looking forward to your response. Thanks!

[simran135]

Thanks @siyan-zhao for raising this with the TRL team.

For reference, this behavior was originally identified and discussed in my original OPSD issue.

In that thread, I reported that _generate_on_policy_outputs_vllm decodes prompts with skip_special_tokens=True before sending them to vLLM. This strips chat-template tokens (e.g. <|im_start|>, <|im_end|>), meaning vLLM receives plain text rather than the original templated prompt.

As a consequence:

• student rollouts become raw text continuations instead of chat-mode completions

• template-controlled behaviors (e.g. thinking modes in chat models) may not activate during generation

• the prompt is re-tokenized, which can create a length mismatch relative to the original student_prompt_length

The maintainers confirmed there that this behavior was not intentional and was inherited from TRL’s GOLDTrainer.

Linking the original report and discussion here so the timeline and prior analysis are available. I’ve attached a screenshot from that thread as well for convenience.