Merge branch 'main' into VESPO

Author: casinca

docs/source/grpo_trainer.md

@@ -695,7 +695,8 @@ trainer.train()
 
 Tested with:
 
-- **Qwen3** — e.g., `Qwen/Qwen3-0.6B`
+- [**Qwen3**](https://huggingface.co/collections/Qwen/qwen3) — e.g., `Qwen/Qwen3-0.6B`
+- [**Qwen3.5**](https://huggingface.co/collections/Qwen/qwen35) — e.g., `Qwen/Qwen3.5-2B`
 
 > [!TIP]
 > Compatibility with all LLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub — or better yet, submit a pull request with the required changes.

docs/source/vllm_integration.md

@@ -3,7 +3,7 @@
 This document will guide you through the process of using vLLM with TRL for faster generation in online methods like GRPO and Online DPO. We first summarize a tl;dr on how to use vLLM with TRL, and then we will go into the details of how it works under the hood.
 
 > [!WARNING]
-> TRL currently only supports vLLM versions `0.10.2`, `0.11.0`, `0.11.1`, `0.11.2` and `0.12.0`. Please ensure you have one of these versions installed to avoid compatibility issues.
+> TRL currently only supports vLLM versions from `0.10.2` to `0.14.1`. Please ensure you have a version in this range installed to avoid compatibility issues.
 
 > [!TIP]
 > The following trainers currently support generation with vLLM:
@@ -31,12 +31,12 @@ pip install "trl[vllm]"
 Then run the server on specific GPUs (e.g., GPUs 0-3):
 
 ```sh
-CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 2 --data-parallel-size 2
+CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 4
 ```
 
 Once the server is running, you can use it to generate completions for training. In the example below, we are using the different supported trainers using the vLLM server for generation. The `--tensor-parallel-size` and `--data-parallel-size` arguments control how the model and data are sharded across GPUs.
 
-In this example, we are sharding two copies of the model across 4 GPUs. Increasing data parallelism increases throughput, while increasing tensor parallelism allows for serving larger models. Then, run the training script on different GPUs (e.g., GPUs 4-7) by passing `use_vllm=True` in the training arguments as follows:
+In this example, we shard one model across 4 GPUs with tensor parallelism. Then, run the training script on different GPUs (e.g., GPUs 4-7) by passing `use_vllm=True` in the training arguments as follows:
 
 Sample of a simple `train.py` script:
 
@@ -166,19 +166,15 @@ If you've ever done autoregressive decoder training, you know all the input toke
 When you run for example
 
 ```sh
-CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 1 --data-parallel-size 4
+CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-7B --tensor-parallel-size 4
 ```
 
-the following happens:
+1. vLLM first spawns multiple workers to handle incoming requests in parallel. The number of workers is determined by multiplying the `--tensor-parallel-size` and `--data-parallel-size` values. In this example, it spawns 4 workers (4 × 1).
+Each worker operates independently and processes a chunk of the incoming requests — which are basically the prompts sent to the server for generation.
 
-![vllm](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/vllm-doc.png)
+2. Once the incoming requests (prompts) are distributed across the workers, the model starts generating completions. Internally, the model’s weights are split across multiple GPUs based on the `--tensor-parallel-size` argument — this is how tensor parallelism is handled.
 
-1. vLLM first spawns multiple workers to handle incoming requests in parallel. The number of workers is determined by multiplying the `--tensor-parallel-size` and `--data-parallel-size` values. In this example, it spawns 4 workers (1 × 4).
-Each worker operates independently and processes a chunk of the incoming requests — which are basically the prompts sent to the server for generation. A key point to understand is that these 4 workers are running in parallel, and each one is responsible for handling a subset of the total incoming load.
-
-2. Once the incoming requests (prompts) are distributed across the workers, the model starts generating completions. Internally, the model’s weights are split across multiple GPUs based on the `--tensor-parallel-size` argument — this is how tensor parallelism is handled. Meanwhile, data parallelism (controlled by `--data-parallel-size`) ensures that different sets of requests are processed independently across the workers. In short: tensor parallelism splits the model across GPUs, and data parallelism splits the batch of requests across different model replicas.
-
-3. Although the GPUs process requests independently and in parallel, they still need to communicate with each other. Remember that each GPU handles only a slice of the incoming prompts (for example, with 4 GPUs and 8 prompts using `--data-parallel-size=4`, each GPU processes 2 prompts).
+3. Although the GPUs process requests independently and in parallel, they still need to communicate with each other. Remember that each GPU handles only a slice of the incoming prompts (for example, with 4 GPUs and 8 prompts using `--tensor-parallel-size=4`, each GPU participates in serving the full model).
 This GPU-to-GPU communication is managed efficiently by NVIDIA’s NCCL library. The communication mainly ensures that each GPU gets its correct portion of the incoming requests — it’s lightweight and doesn’t interfere with generation itself.
 Separately, the number of completions to generate per prompt is controlled by the `num_generations` setting in the GRPO config. For instance, if you set `num_generations=2` (like in the picture above), each prompt will have 2 completions. So, with 8 prompts and `num_generations=2`, you would end up with 16 completions total — regardless of the number of GPUs or parallelism settings.
 
@@ -224,7 +220,9 @@ options:
   --tensor_parallel_size TENSOR_PARALLEL_SIZE, --tensor-parallel-size TENSOR_PARALLEL_SIZE
                         Number of tensor parallel workers to use. (default: 1)
   --data_parallel_size DATA_PARALLEL_SIZE, --data-parallel-size DATA_PARALLEL_SIZE
-                        Number of data parallel workers to use. (default: 1)
+                        Number of data parallel workers to use. For dense models, keep this at 1. Starting from vLLM `0.14.0`, setting
+                        this above `1` for dense models is no longer supported/useful and will error out (see vLLM PR #30739).
+                        (default: 1)
   --host HOST           Host address to run the server on. (default: 0.0.0.0)
   --port PORT           Port to run the server on. (default: 8000)
   --gpu_memory_utilization GPU_MEMORY_UTILIZATION, --gpu-memory-utilization GPU_MEMORY_UTILIZATION
@@ -259,20 +257,8 @@ options:
 ![tp dp throughput 8 gpus](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/tp_dp_throughput_8_gpus.png)
 ![tp dp throughput 4 gpus](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/tp_dp_throughput_4_gpus.png)
 
-First and foremost, always remember that the optimal setup depends on:
-
-- The model size
-- The number of GPUs you have
-- The GPU memory size
-- The batch size you are using
-- The number of requests you are sending to the server (prompts)
-- The `max_model_len` you are using (this is the max length of the input sequence that the model can process, a.k.a. the context window size)
-- The number of completions you are generating for each request (`num_generations`)
-
-Given these factors, our experiments on the Qwen model family (3B, 7B, 14B, 32B) using 8 H100 GPUs show that:
-
-- For reasonable-sized models (3B–14B) and a moderate context window (`max_len < 8k`), using full capacity for data parallelism gives better throughput. The setup `(tp=1, dp=8)` yields the best results.
-- For larger models (32B) and longer context windows (`max_len > 8k`), a smaller DP size combined with some model-side parallelism performs better. For example, `(tp=2, dp=4)` is a good setup for 32B models with a larger context window.
+> [!WARNING]
+> The benchmark plots above were collected with older vLLM versions. Starting with [vLLM PR #30739](https://github.com/vllm-project/vllm/pull/30739) (released in `0.14.0`), offline data parallel scaling for non-MoE (dense) models is no longer supported. To follow the latest recommendations, do not scale DP for non-MoE models.
 
 ### vLLM with Transformers Backend
 

pyproject.toml

@@ -83,7 +83,7 @@ test = [
     "pytest"
 ]
 vllm = [
-    "vllm>=0.10.2,<0.13.0",
+    "vllm>=0.10.2,<=0.14.1",
     "fastapi",
     "pydantic",
     "requests",

scripts/generate_tiny_models.py

@@ -71,6 +71,8 @@
     Qwen2ForSequenceClassification,
     Qwen2VLConfig,
     Qwen2VLForConditionalGeneration,
+    Qwen3_5Config,
+    Qwen3_5ForConditionalGeneration,
     Qwen3Config,
     Qwen3ForCausalLM,
     Qwen3ForSequenceClassification,
@@ -325,9 +327,10 @@ def init_weights_tiny_model(model):
     ("Qwen/Qwen2-VL-2B-Instruct", Qwen2VLForConditionalGeneration, torch.bfloat16),
     ("Qwen/Qwen2.5-VL-3B-Instruct", Qwen2_5_VLForConditionalGeneration, torch.bfloat16),
     ("Qwen/Qwen3-VL-2B-Instruct", Qwen3VLForConditionalGeneration, torch.bfloat16),
+    ("Qwen/Qwen3.5-0.8B", Qwen3_5ForConditionalGeneration, torch.bfloat16),
 ]:
     processor = AutoProcessor.from_pretrained(model_id)
-    generation_config = GenerationConfig.from_pretrained(model_id)
+    generation_config = GenerationConfig.from_pretrained(model_id) if model_id != "Qwen/Qwen3.5-0.8B" else None
 
     text_config = {
         "num_hidden_layers": 2,
@@ -371,13 +374,36 @@ def init_weights_tiny_model(model):
         vision_config["depth"] = 2
         vision_config["out_hidden_size"] = 16
 
+    if issubclass(model_class.config_class, Qwen3_5Config):
+        # For tiny layer counts, default `layer_types` can end up with no full-attention layers (e.g. 2 layers and
+        # default interval 4), which breaks Qwen3.5 dynamic cache logic. Keep one full-attention layer at the end.
+        text_config["layer_types"] = ["linear_attention", "full_attention"]
+        text_config["full_attention_interval"] = 2
+        # Qwen3.5-VL vision config expects `depth`/`num_heads`, not `num_hidden_layers`/`num_attention_heads`.
+        vision_config.pop("num_hidden_layers", None)
+        vision_config.pop("num_attention_heads", None)
+        vision_config.pop("num_key_value_heads", None)
+        vision_config.pop("embed_dim", None)
+        vision_config["depth"] = 2
+        vision_config["num_heads"] = 4
+        vision_config["intermediate_size"] = 32
+        vision_config["out_hidden_size"] = 16
+
     if model_id == "llava-hf/llava-v1.6-mistral-7b-hf":
         # Hotfix: llava-hf/llava-v1.6-mistral-7b-hf mistakesly sets text_config.dtype to "bfloat16".
         # See https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/discussions/46
         text_config["dtype"] = None
 
     config = AutoConfig.from_pretrained(model_id, text_config=text_config, vision_config=vision_config, **kwargs)
     model = model_class(config).to(dtype=dtype)
+
+    if issubclass(model_class.config_class, Qwen3_5Config):
+        # Qwen3.5 models has some weights in float32, to mirror this in the tiny model we need to convert them to float32 manually.
+        for layer in model.model.language_model.layers:
+            if hasattr(layer, "linear_attn"):  # applies to linear attention layers only
+                layer.linear_attn.A_log.data = layer.linear_attn.A_log.data.float()
+                layer.linear_attn.norm.weight.data = layer.linear_attn.norm.weight.data.float()
+
     push_to_hub(model, processor, generation_config, "tiny")
 
 # PEFT models

tests/test_chat_template_utils.py

@@ -114,6 +114,7 @@ def test_clone_with_sequence_classification_model(self):
     "tokenizer_name",
     [
         pytest.param("trl-internal-testing/tiny-Qwen3MoeForSequenceClassification", id="qwen3"),
+        pytest.param("trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration", id="qwen35"),
     ],
 )
 @pytest.mark.xfail(
@@ -217,6 +218,14 @@ def test_non_prefix_preserving_template(self):
     "tokenizer_name",
     [
         pytest.param("trl-internal-testing/tiny-Qwen3MoeForSequenceClassification", id="qwen3"),
+        pytest.param(
+            "trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration",
+            id="qwen35",
+            marks=pytest.mark.skipif(
+                Version(transformers.__version__) < Version("5.0.0"),
+                reason="Qwen3.5 tokenizer requires transformers>=5.0.0",
+            ),
+        ),
     ],
 )
 class TestGetTrainingChatTemplate:
@@ -302,21 +311,6 @@ def test_behavior_unchanged_assistant_with_tool_calls(self, tokenizer_name):
         after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)
         assert before == after
 
-    def test_behavior_unchanged_assistant_with_tool_calls_with_string_arguments(self, tokenizer_name):
-        tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
-        messages = [
-            {"role": "user", "content": "Multiply 3 by 4."},
-            {
-                "role": "assistant",
-                "content": "I will call a tool.",
-                "tool_calls": [{"name": "multiply", "arguments": '{"a": 3, "b": 4}'}],
-            },
-        ]
-        before = tokenizer.apply_chat_template(messages, tokenize=False)
-        new_chat_template = get_training_chat_template(tokenizer)
-        after = tokenizer.apply_chat_template(messages, tokenize=False, chat_template=new_chat_template)
-        assert before == after
-
     def test_behavior_unchanged_with_tools_with_and_without_system_message(self, tokenizer_name):
         tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
         tools = [
@@ -388,6 +382,7 @@ def test_behavior_unchanged_generation_prompt_with_enable_thinking_false(self, t
     "tokenizer_name",
     [
         pytest.param("trl-internal-testing/tiny-Qwen3MoeForSequenceClassification", id="qwen3"),
+        pytest.param("trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration", id="qwen35"),
     ],
 )
 @pytest.mark.xfail(
@@ -417,7 +412,11 @@ def test_parse_response_with_reasoning_content(self, tokenizer_name):
             {"role": "user", "content": "What is 3*4?"},
             {"role": "assistant", "reasoning_content": "Hmmm.", "content": "12"},
         ]
-        prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids
+        # enable_thinking=True is required here because for Qwen3.5, the thinking is disabled by default for the
+        # generation prompt.
+        prefix = tokenizer.apply_chat_template(
+            messages[:1], add_generation_prompt=True, enable_thinking=True
+        ).input_ids
         text = tokenizer.apply_chat_template(messages).input_ids
         response = text[len(prefix) :]
         parsed = parse_response(tokenizer, response)
@@ -451,6 +450,20 @@ def test_parse_response_tool_call_with_content(self, tokenizer_name):
         parsed = parse_response(tokenizer, response)
         assert parsed == messages[-1]
 
+    def test_parse_response_tool_call_without_arguments(self, tokenizer_name):
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
+        tokenizer = add_response_schema(tokenizer)
+        tool_calls = [{"type": "function", "function": {"name": "ping", "arguments": {}}}]
+        messages = [
+            {"role": "user", "content": "Ping the service."},
+            {"role": "assistant", "tool_calls": tool_calls},
+        ]
+        prefix = tokenizer.apply_chat_template(messages[:1], add_generation_prompt=True).input_ids
+        text = tokenizer.apply_chat_template(messages).input_ids
+        response = text[len(prefix) :]
+        parsed = parse_response(tokenizer, response)
+        assert parsed == {"role": "assistant", "content": "", "tool_calls": tool_calls}
+
     def test_parse_response_multiple_tool_calls(self, tokenizer_name):
         tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
         tokenizer = add_response_schema(tokenizer)

tests/test_data_utils.py

@@ -481,6 +481,13 @@ class TestApplyChatTemplate(TrlTestCase):
         "trl-internal-testing/tiny-Phi3ForCausalLM",
         "trl-internal-testing/tiny-Qwen2ForCausalLM-2.5",
         "trl-internal-testing/tiny-Qwen3ForCausalLM",
+        pytest.param(
+            "trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration",
+            marks=pytest.mark.skipif(
+                Version(transformers.__version__) < Version("5.0.0"),
+                reason="Qwen3.5 tokenizer requires transformers>=5.0.0",
+            ),
+        ),
     ]
 
     conversational_examples = [

tests/test_dpo_trainer.py

@@ -1009,6 +1009,13 @@ def test_tag_added_peft(self):
                     ),
                 ],
             ),
+            pytest.param(
+                "trl-internal-testing/tiny-Qwen3_5ForConditionalGeneration",
+                marks=pytest.mark.skipif(
+                    Version(transformers.__version__) < Version("5.2.0"),
+                    reason="Qwen3.5 models were introduced in transformers-5.2.0",
+                ),
+            ),
         ],
     )
     @require_vision