Squashed commit of the following:

commit 9925469
Author: Pramodith Ballapuram <[email protected]>
Date:   Wed Oct 29 22:09:47 2025 +0000

    Support chat_template_kwargs (#4350)

    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 4e9ab9f
Author: Kashif Rasul <[email protected]>
Date:   Wed Oct 29 18:20:15 2025 +0100

    👑 [experimental] GOLD Trainer (#4349)

    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit b82a8f4
Author: Kamran Bigdely <[email protected]>
Date:   Wed Oct 29 10:16:22 2025 -0700

    🔥 docs: Add RapidFire AI integration guide (#4340)

    Co-authored-by: kamran bigdely <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 29fb69f
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 17:45:06 2025 +0100

    Align make test_experimental with make test (#4371)

commit ac6cea8
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 17:25:16 2025 +0100

    Fix add_generation_prompt arg for paged transformers in GRPO and RLOO trainers (#4370)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 1e39eb6
Author: Taha Yassine <[email protected]>
Date:   Wed Oct 29 16:59:49 2025 +0100

    Add support for Trackio completions logging in GRPOTrainer (#4359)

    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 97830a3
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 11:13:54 2025 +0100

    Replace deprecated list with tuple indexing in PPOTrainer (#4356)

commit d275418
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 11:13:33 2025 +0100

    Remove ignored max_length parameter from PRMTrainer data collator (#4355)

commit 61bf96c
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 11:13:04 2025 +0100

    Move tests of BCO trainer args to tests/experimental (#4354)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit b8f23ef
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 08:00:50 2025 +0100

    Replace deprecated AutoModelForVision2Seq with AutoModelForImageTextToText (#4353)

commit f8073cb
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 07:53:13 2025 +0100

    Implement CI test workflow for experimental module (#4330)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 55854c8
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 07:42:46 2025 +0100

    Move tests of experimental GRPO with replay buffer to tests/experimental (#4329)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 4352074
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 29 07:41:43 2025 +0100

    Use explicit tiny-Qwen2_5_VL model_id parameter in CI tests (#4325)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 928f589
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 28 18:12:24 2025 -0600

    Fix: `add_generation_prompt=True` for conversational only (#4362)

commit b0889d2
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 28 18:00:27 2025 -0600

    Add `add_generation_prompt` to processor_kwargs in GRPO and RLOO trainer (#4361)

commit a9d33d0
Author: kaixuanliu <[email protected]>
Date:   Wed Oct 29 05:13:59 2025 +0800

    fix CI issue for vlm_gemma_3n model (#4278)

    Signed-off-by: Liu, Kaixuan <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Albert Villanova del Moral <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 34fdb61
Author: Sergio Paniego Blanco <[email protected]>
Date:   Tue Oct 28 20:51:22 2025 +0100

    Fixed links inside Tips in docs (#4360)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit a23e91c
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 28 19:48:42 2025 +0000

    Add missing license in `tests/experimental/__init__.py`

commit 5e691d1
Author: Albert Villanova del Moral <[email protected]>
Date:   Mon Oct 27 22:01:31 2025 +0100

    Fix GRPO and RLOO trainers for continuous batching (#4348)

commit fa644b1
Author: Kashif Rasul <[email protected]>
Date:   Mon Oct 27 14:01:34 2025 +0100

    [vllm] update comment about communication group host ip (#4337)

commit fda88c6
Author: Sergio Paniego Blanco <[email protected]>
Date:   Mon Oct 27 10:29:24 2025 +0100

    Added custom `prepare_model_for_kbit_training` to save VRAM (#4335)

    Co-authored-by: Kashif Rasul <[email protected]>

commit 2a138c7
Author: Sergio Paniego Blanco <[email protected]>
Date:   Mon Oct 27 10:26:09 2025 +0100

    Update Reducing Memory Consumption guide with more details (#4332)

    Co-authored-by: Kashif Rasul <[email protected]>

commit 05a1feb
Author: Quentin Gallouédec <[email protected]>
Date:   Fri Oct 24 11:48:30 2025 -0700

    🗞️ Update "What's New" (#4338)

commit d8543c0
Author: Sergio Paniego Blanco <[email protected]>
Date:   Fri Oct 24 11:27:25 2025 +0200

    Add OpenEnv blog to landing (#4333)

commit 23c0062
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 24 09:48:37 2025 +0200

    Hotfix: Fall back to config.text_config._name_or_path if missing config._name_or_path (#4324)

commit 47b1aa7
Author: Albert Villanova del Moral <[email protected]>
Date:   Thu Oct 23 12:04:46 2025 +0200

    Move BCO tests to tests/experimental (#4326)

commit a4872d9
Author: Sergio Paniego Blanco <[email protected]>
Date:   Thu Oct 23 11:42:13 2025 +0200

    Update OpenEnv docs (#4328)

commit 3f66564
Author: Sergio Paniego Blanco <[email protected]>
Date:   Thu Oct 23 10:45:23 2025 +0200

    Highlight OpenEnv in landing docs (#4327)

commit 9b80e33
Author: Quentin Gallouédec <[email protected]>
Date:   Thu Oct 23 07:45:54 2025 +0000

    Update documentation openenv

commit 2819a8f
Author: lewtun <[email protected]>
Date:   Thu Oct 23 09:36:35 2025 +0200

    🕹️ Add rollout function for OpenEnv integration (#4310)

    Co-authored-by: Kashif Rasul <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit e1c87e3
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 22 18:21:44 2025 +0200

    Fix attn_implementation name in OnlineDPO for transformers v5 (#4322)

commit 7c547a3
Author: Sergio Paniego Blanco <[email protected]>
Date:   Wed Oct 22 09:16:25 2025 +0200

    Add notebooks to Examples docs and restructure (#4317)

    Co-authored-by: Kashif Rasul <[email protected]>

commit bfd6f49
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 22 08:43:31 2025 +0200

    Replace unittest skipTest from transformers with pytest.skip (#4297)

commit 712f6a9
Author: Hsiang-Yu Tsou <[email protected]>
Date:   Wed Oct 22 12:04:13 2025 +0800

    💤 Switch to sleep level=2 and split wake-ups in GRPO and RLOO trainers (#4296)

commit 1382e56
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 21 15:41:42 2025 -0700

    🧺 [5/N] Refactor `_generate` in GRPO/RLOO: Insert images in the prompt (#4155)

commit cb9bc2a
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 21 12:51:48 2025 -0700

    🚚 Move BCO to `trl.experimental` (#4312)

commit 475c732
Author: Sergio Paniego Blanco <[email protected]>
Date:   Tue Oct 21 17:17:07 2025 +0200

    Update notebooks README with latest additions (#4316)

commit 0dc4d53
Author: Albert Villanova del Moral <[email protected]>
Date:   Tue Oct 21 15:54:59 2025 +0200

    Remove parameterized as test extra dependency (#4315)

commit e2ab435
Author: Kashif Rasul <[email protected]>
Date:   Tue Oct 21 12:34:18 2025 +0200

    [Activation-checkpointing] add tensor dedup and param offloading (#4247)

    Co-authored-by: Sergio Paniego Blanco <[email protected]>

commit 46a53cd
Author: Albert Villanova del Moral <[email protected]>
Date:   Tue Oct 21 10:23:00 2025 +0200

    Filter expected setup_chat_format deprecation warning in CI (#4306)

commit 6105040
Author: Albert Villanova del Moral <[email protected]>
Date:   Tue Oct 21 10:22:42 2025 +0200

    Silence TRL experimental warnings in CI (#4307)

commit 5eae44a
Author: Quentin Gallouédec <[email protected]>
Date:   Mon Oct 20 13:27:21 2025 -0600

    ⚰️ Remove deprecated (#4301)

commit 28bba8c
Author: Sergio Paniego Blanco <[email protected]>
Date:   Mon Oct 20 11:24:54 2025 +0200

    Added SFT LoRA notebook (#4244)

    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Albert Villanova del Moral <[email protected]>

commit 2f1802b
Author: Albert Villanova del Moral <[email protected]>
Date:   Mon Oct 20 08:03:48 2025 +0200

    Fix missing CI slow tests: ImportError: vLLM is not installed (#4304)

commit e0eec05
Author: Quentin Gallouédec <[email protected]>
Date:   Fri Oct 17 15:36:13 2025 -0600

    🧺 [4/N] Refactor `_generate` in GRPO/RLOO: Move `forward_kwargs` outside generation method (#4154)

    Co-authored-by: Albert Villanova del Moral <[email protected]>
    Co-authored-by: YonatanGideoni <[email protected]>
    Co-authored-by: burtenshaw <[email protected]>
    Co-authored-by: sergiopaniego <[email protected]>
    Co-authored-by: lewtun <[email protected]>
    Co-authored-by: Kashif Rasul <[email protected]>

commit f4c554d
Author: Sergio Paniego Blanco <[email protected]>
Date:   Fri Oct 17 16:06:40 2025 +0200

    Update links to docs in README to latest packaged version (#4084)

commit a932e27
Author: Quentin Gallouédec <[email protected]>
Date:   Wed Oct 15 18:11:52 2025 -0600

    ⬆️ Bump dev version (#4293)

commit 04fd120
Author: Quentin Gallouédec <[email protected]>
Date:   Wed Oct 15 18:10:10 2025 -0600

    Release: v0.24 (#4292)

commit 19d2f97
Author: Quentin Gallouédec <[email protected]>
Date:   Wed Oct 15 18:06:34 2025 -0600

    Deprecate `BestOfNSampler` (#4291)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>
    Co-authored-by: Behrooz Azarkhalili <[email protected]>

commit 31caf64
Author: Behrooz Azarkhalili <[email protected]>
Date:   Wed Oct 15 17:01:50 2025 -0700

    Remove unused commands directory (#4258)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>

commit 8e2d551
Author: Pramodith Ballapuram <[email protected]>
Date:   Thu Oct 16 01:01:07 2025 +0100

    Add accuracy reward (#4270)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 94aac4a
Author: Behrooz Azarkhalili <[email protected]>
Date:   Wed Oct 15 16:49:04 2025 -0700

    Remove how_to_train.md: outdated training FAQ (#4267)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>

commit 26b7c25
Author: Alexander Weers <[email protected]>
Date:   Thu Oct 16 01:33:35 2025 +0200

    Add support for `token_type_ids` in `DPOTrainer` (#4285)

    Co-authored-by: Quentin Gallouédec <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit aa25c26
Author: Behrooz Azarkhalili <[email protected]>
Date:   Wed Oct 15 14:13:27 2025 -0700

    Remove using_llama_models.md: outdated Llama2-specific documentation (#4268)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>

commit 93c7d88
Author: Behrooz Azarkhalili <[email protected]>
Date:   Wed Oct 15 14:12:32 2025 -0700

    Remove logging.md: trainer-specific metrics documentation (#4269)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>

commit c7c041e
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 15 18:15:36 2025 +0200

    Fix CI slow tests: ImportError: vLLM is not installed (#4287)

commit ef40c04
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 15 18:15:28 2025 +0200

    Replace unittest skipTest with pytest.skip (#4263)

commit 7e0adbc
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 15 18:14:49 2025 +0200

    Fix CI dev test TypeError: unexpected keyword argument 'load_in_4bit' (#4262)

commit 773afd9
Author: Quentin Gallouédec <[email protected]>
Date:   Wed Oct 15 09:39:17 2025 -0600

    💰 `RichProgressCallback` enhancement (#4245)

commit 966b397
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 15 16:11:11 2025 +0200

    Fix CI slow test OSError: You are trying to access a gated repo (#4283)

commit 927cf6b
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 15 10:39:12 2025 +0200

    Fix docstrings with Sphinx 'deprecated' directive (#4279)

commit 56cb6cc
Author: Sergio Paniego Blanco <[email protected]>
Date:   Tue Oct 14 18:51:17 2025 +0200

    Fix typo in Colab link (#4276)

commit 49c8f14
Author: Sergio Paniego Blanco <[email protected]>
Date:   Tue Oct 14 18:45:01 2025 +0200

    Add Qwen3-VL notebooks (SFT, GRPO) (#4275)

    Co-authored-by: Copilot <[email protected]>

commit cefbacb
Author: Albert Villanova del Moral <[email protected]>
Date:   Tue Oct 14 12:13:15 2025 +0200

    Fix style with make precommit (#4265)

commit fae245a
Author: Albert Villanova del Moral <[email protected]>
Date:   Tue Oct 14 12:12:03 2025 +0200

    Use FutureWarning instead of DeprecationWarning (#4266)

commit 2aa9506
Author: Albert Villanova del Moral <[email protected]>
Date:   Mon Oct 13 13:40:24 2025 +0200

    Fix docstring interlinks (#4221)

commit d6eeb29
Author: Albert Villanova del Moral <[email protected]>
Date:   Mon Oct 13 11:06:09 2025 +0200

    Raise deprecation warning for Python 3.9 (#4226)

commit 1684ef2
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 17:41:24 2025 +0200

    Fix Python version check for skipping tests on Python 3.13.8 (#4246)

commit aab21eb
Author: Carlos Miguel Patiño <[email protected]>
Date:   Fri Oct 10 17:39:29 2025 +0200

    Include `chat_template_kwargs` in `apply_chat_template` (#4233)

    Co-authored-by: Quentin Gallouédec <[email protected]>

commit b997a31
Author: Kashif Rasul <[email protected]>
Date:   Fri Oct 10 17:21:01 2025 +0200

    [Online-DPO] fix the completion_len == max_new_tokens crash (#4193)

    Co-authored-by: Albert Villanova del Moral <[email protected]>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 86d1963
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 17:19:53 2025 +0200

    Fix CI slow test AttributeError: 'TestSFTTrainerSlow' object has no attribute 'addCleanup' (#4255)

commit 039d526
Author: Behrooz Azarkhalili <[email protected]>
Date:   Fri Oct 10 08:16:18 2025 -0700

    Deprecate unused dataset_formatting module (#4242)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit bcd059a
Author: Behrooz Azarkhalili <[email protected]>
Date:   Fri Oct 10 08:15:47 2025 -0700

    Remove obsolete research_projects directory (#4243)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>
    Co-authored-by: Quentin Gallouédec <[email protected]>

commit 0e57b4a
Author: Quentin Gallouédec <[email protected]>
Date:   Fri Oct 10 10:02:11 2025 -0500

    🧺 [3/N] Refactor `_generate` in GRPO/RLOO: Rely on generator for prompt truncation (#4153)

commit 98488e0
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 16:37:02 2025 +0200

    Fix CI slow test ValueError: Unknown loss type: dapo (#4254)

commit f45e865
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 16:13:22 2025 +0200

    Fix CI ImportError for 'require_torch_gpu_if_bnb_not_multi_backend_enabled' (#4253)

commit f582792
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 16:12:15 2025 +0200

    Install peft from main for CI tests with dev dependencies (#4250)

commit f853e09
Author: Albert Villanova del Moral <[email protected]>
Date:   Fri Oct 10 09:49:45 2025 +0200

    Fix CI CUDA out of memory errors by improving GPU memory management (#4238)

commit 803ec0d
Author: Wang, Yi <[email protected]>
Date:   Fri Oct 10 15:28:34 2025 +0800

    Fix CI slow test ValueError: Backward pass should have cleared tracker of all tensors (#4236)

    Signed-off-by: Wang, Yi A <[email protected]>
    Co-authored-by: Kashif Rasul <[email protected]>

commit 7a0a615
Author: Quentin Gallouédec <[email protected]>
Date:   Thu Oct 9 17:05:36 2025 -0600

    Warnings pointing to RFC (#4224)

commit c38cb69
Author: Quentin Gallouédec <[email protected]>
Date:   Thu Oct 9 12:49:44 2025 -0600

    🧘 Enhance markdown style (#4235)

commit 68ef15c
Author: Behrooz Azarkhalili <[email protected]>
Date:   Thu Oct 9 09:18:48 2025 -0700

    Remove unused log_example_reports.py script (#4241)

    Co-authored-by: behroozazarkhalili <ermiaazarkhalili>

commit 3dd7fc2
Author: Albert Villanova del Moral <[email protected]>
Date:   Thu Oct 9 15:46:41 2025 +0200

    Fix CI IndentationError for Python 3.13.8 (#4240)

commit 51ced65
Author: Albert Villanova del Moral <[email protected]>
Date:   Thu Oct 9 09:35:08 2025 +0200

    Replace setup with pyproject in CI tests paths (#4230)

commit 4bb883a
Author: Albert Villanova del Moral <[email protected]>
Date:   Thu Oct 9 08:09:15 2025 +0200

    Update CI Docker image to pytorch/pytorch:2.8.0 (#4232)

commit f784632
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 8 21:30:54 2025 +0200

    Remove unused Path import in __init__.py (#4227)

commit a944890
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 8 21:21:21 2025 +0200

    Fix callable annotations (#4216)

commit 521db35
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 8 21:18:41 2025 +0200

    Fix CI unittest asserts (#4234)

commit e2c97a8
Author: Albert Villanova del Moral <[email protected]>
Date:   Wed Oct 8 18:14:23 2025 +0200

    Exclude vllm dependencies from dev extra (#4229)

commit d1d0407
Author: Quentin Gallouédec <[email protected]>
Date:   Wed Oct 8 09:34:48 2025 -0600

    🏷️ Account for `token_type_ids` in `DataCollatorForVisionLanguageModeling` (#4190)

commit 824ff8c
Author: Sergio Paniego Blanco <[email protected]>
Date:   Wed Oct 8 12:59:04 2025 +0200

    Add Efficient Online Training with GRPO and vLLM in TRL to community tutorials (#4219)

commit f15399d
Author: Pramodith Ballapuram <[email protected]>
Date:   Wed Oct 8 09:42:19 2025 +0100

    Fix entropy and accuracy calculation for prompt_tuning techniques. (#4196)

commit cc578b6
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 7 12:11:34 2025 -0600

    🧺 [2/N] Refactor `_generate` in GRPO/RLOO: Use `prompt_ids` from generation (#4152)

commit 30cf68a
Author: Quentin Gallouédec <[email protected]>
Date:   Tue Oct 7 10:21:10 2025 -0600

    🎨 Support mixing image+text and text-only examples (#4203)

    Co-authored-by: Sergio Paniego Blanco <[email protected]>

Author: qgallouedec

.github/workflows/slow-tests.yml

@@ -11,6 +11,7 @@ env:
   RUN_SLOW: "yes"
   IS_GITHUB_CI: "1"
   SLACK_API_TOKEN: ${{ secrets.SLACK_CIFEEDBACK_BOT_TOKEN }}
+  TRL_EXPERIMENTAL_SILENCE: 1
 
 jobs:
   run_all_tests_single_gpu:
@@ -46,7 +47,7 @@ jobs:
         run: |
           source .venv/bin/activate
           uv pip install ".[dev]"
-          uv pip install pytest-reportlog parameterized
+          uv pip install pytest-reportlog
 
       - name: Run slow SFT tests on single GPU
         if: always()
@@ -68,7 +69,7 @@ jobs:
       CUDA_VISIBLE_DEVICES: "0,1"
       TEST_TYPE: "multi_gpu"
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all --shm-size "16gb"
     defaults:
       run:
@@ -94,27 +95,18 @@ jobs:
         run: |
           source .venv/bin/activate
           uv pip install ".[dev]"
-          uv pip install pytest-reportlog parameterized
+          uv pip install pytest-reportlog
 
       - name: Run slow SFT tests on Multi GPU
         if: always()
         run: |
           source .venv/bin/activate
           make slow_tests
 
-      - name: Run end-to-end examples tests on multi GPU
-        if: always()
-        run: |
-          source .venv/bin/activate
-          uv pip install deepspeed
-          make test_examples
-
       - name: Generate Reports
         if: always()
         run: |
           source .venv/bin/activate
           uv pip install slack_sdk tabulate
           python scripts/log_reports.py >> $GITHUB_STEP_SUMMARY
-          python scripts/log_example_reports.py --text_file_name temp_results_sft_tests.txt >> $GITHUB_STEP_SUMMARY
-          python scripts/log_example_reports.py --text_file_name temp_results_dpo_tests.txt >> $GITHUB_STEP_SUMMARY
           rm *.txt

.github/workflows/tests-experimental.yml

@@ -0,0 +1,70 @@
+name: Tests (experimental)
+
+on:
+  pull_request:
+    paths:
+      # Run only when relevant files are modified
+      - "trl/experimental/**"
+      - "tests/experimental/**"
+
+env:
+  TQDM_DISABLE: 1
+  PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
+  TRL_EXPERIMENTAL_SILENCE: 1
+
+jobs:
+  check_code_quality:
+    name: Check code quality
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.draft == false
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.13
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.13
+      - uses: pre-commit/[email protected]
+        with:
+          extra_args: --all-files
+
+  tests:
+    name: Tests (experimental)
+    runs-on:
+      group: aws-g4dn-2xlarge
+    container:
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
+      options: --gpus all
+    defaults:
+      run:
+        shell: bash
+    steps:
+      - name: Git checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.13
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.13
+
+      - name: Install Make and Git
+        run: |
+          apt-get update && apt-get install -y make git curl
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+
+      - name: Create Python virtual environment
+        run: |
+          uv venv
+          uv pip install --upgrade setuptools wheel
+
+      - name: Install dependencies
+        run: |
+          source .venv/bin/activate
+          uv pip install ".[dev]"
+
+      - name: Test with pytest
+        run: |
+          source .venv/bin/activate
+          make test_experimental

.github/workflows/tests.yml

@@ -11,11 +11,15 @@ on:
       - "scripts/**.py"
       - "tests/**.py"
       - "trl/**.py"
-      - "setup.py"
+      - "pyproject.toml"
+      # Exclude if only experimental code/tests
+      - "!trl/experimental/**"
+      - "!tests/experimental/**"
 
 env:
   TQDM_DISABLE: 1
   CI_SLACK_CHANNEL: ${{ secrets.CI_PUSH_MAIN_CHANNEL }}
+  PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
 
 jobs:
   check_code_quality:
@@ -41,7 +45,7 @@ jobs:
     runs-on:
       group: aws-g4dn-2xlarge
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all
     defaults:
       run:
@@ -93,7 +97,7 @@ jobs:
     runs-on:
       group: aws-g4dn-2xlarge
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all
     defaults:
       run:
@@ -128,7 +132,7 @@ jobs:
           uv pip install -U git+https://github.com/huggingface/accelerate.git
           uv pip install -U git+https://github.com/huggingface/datasets.git
           uv pip install -U git+https://github.com/huggingface/transformers.git
-          
+          uv pip install -U git+https://github.com/huggingface/peft.git
 
       - name: Test with pytest
         run: |
@@ -149,7 +153,7 @@ jobs:
     runs-on:
       group: aws-g4dn-2xlarge
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all
     defaults:
       run:
@@ -201,7 +205,7 @@ jobs:
     runs-on:
       group: aws-g4dn-2xlarge
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all
     defaults:
       run:

.github/workflows/tests_latest.yml

@@ -9,22 +9,23 @@ on:
 env:
   TQDM_DISABLE: 1
   CI_SLACK_CHANNEL: ${{ secrets.CI_PUSH_MAIN_CHANNEL }}
+  TRL_EXPERIMENTAL_SILENCE: 1
 
 jobs:
   tests:
     name: Tests latest TRL release with dev dependencies
     runs-on:
       group: aws-g4dn-2xlarge
     container:
-      image: pytorch/pytorch:2.6.0-cuda12.6-cudnn9-devel
+      image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel
       options: --gpus all
     defaults:
       run:
         shell: bash
     steps:
       - name: Git checkout
         uses: actions/checkout@v4
-        with: { ref: v0.23-release }
+        with: { ref: v0.24-release }
 
       - name: Set up Python 3.12
         uses: actions/setup-python@v5

CITATION.cff

@@ -31,4 +31,4 @@ keywords:
   - pytorch
   - transformers
 license: Apache-2.0
-version: "0.23"
+version: "0.24"