[https://nvbugs/5823783][fix] Fix multi-node trust_remote_code hang i… #11383

JunyiXu-nv · 2026-02-09T09:25:54Z

…ssue

Summary by CodeRabbit

Release Notes

Bug Fixes
- Improved tokenizer serialization support when using remote code, ensuring better compatibility across distributed environments and preventing potential serialization failures in multi-node setups.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

JunyiXu-nv · 2026-02-09T09:26:17Z

/bot run

coderabbitai · 2026-02-09T09:29:38Z

📝 Walkthrough

Walkthrough

Introduced cloudpickle-based serialization support for HuggingFace tokenizers to enable cross-node distributed processing. Added helper functions for reconstructing serialized tokenizers and registering dynamic transformer modules when trust_remote_code is enabled.

Changes

Cohort / File(s)	Summary
Tokenizer Serialization Enhancement `tensorrt_llm/tokenizer/tokenizer.py`	Added `__reduce__` method to `TransformersTokenizer` for cloudpickle by-value serialization, `_reconstruct_transformers_tokenizer()` helper for deserialization, and `maybe_register_transformers_modules_by_value()` to register dynamic modules. Enhanced `load_hf_tokenizer()` to invoke module registration when `trust_remote_code=True`.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant load_hf_tokenizer
    participant TransformersTokenizer
    participant Cloudpickle
    participant HFModules as HF Dynamic Modules

    Client->>load_hf_tokenizer: load_hf_tokenizer(trust_remote_code=True)
    load_hf_tokenizer->>load_hf_tokenizer: maybe_register_transformers_modules_by_value()
    load_hf_tokenizer->>HFModules: Register dynamic modules
    load_hf_tokenizer->>TransformersTokenizer: from_pretrained()
    TransformersTokenizer-->>load_hf_tokenizer: Return tokenizer instance
    Client->>Cloudpickle: pickle(tokenizer)
    Cloudpickle->>TransformersTokenizer: __reduce__()
    TransformersTokenizer-->>Cloudpickle: Return (reconstruct_fn, args)
    Cloudpickle-->>Client: Serialized bytes
    Client->>Cloudpickle: unpickle(bytes)
    Cloudpickle->>Cloudpickle: _reconstruct_transformers_tokenizer(bytes)
    Cloudpickle-->>Client: Deserialized tokenizer

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	PR description lacks critical details: no explanation of the bug, root cause, or implementation approach; all required sections are empty placeholders.	Add a clear description of the multi-node hang issue, explain why it occurs with trust_remote_code, describe the solution, list test cases covering the fix, and verify the PR checklist items are appropriately addressed.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title correctly follows the repository template with NVBugs ID and [fix] type, and clearly indicates the fix addresses a multi-node trust_remote_code hang issue.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-02-09T09:32:25Z

PR_Github #35313 [ run ] triggered by Bot. Commit: 3d51f1f

tensorrt-cicd · 2026-02-09T09:53:49Z

PR_Github #35313 [ run ] completed with state FAILURE. Commit: 3d51f1f
/LLM/release-1.2/L0_MergeRequest_PR pipeline #339 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

JunyiXu-nv · 2026-02-09T11:20:02Z

/bot run

tensorrt-cicd · 2026-02-09T11:26:12Z

PR_Github #35327 [ run ] triggered by Bot. Commit: 3d51f1f

tensorrt-cicd · 2026-02-09T12:08:15Z

PR_Github #35327 [ run ] completed with state SUCCESS. Commit: 3d51f1f
/LLM/release-1.2/L0_MergeRequest_PR pipeline #340 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

litaotju · 2026-02-12T04:44:49Z

tensorrt_llm/tokenizer/tokenizer.py

+        #
+        # See: https://github.com/vllm-project/vllm/pull/6751
+        try:
+            import cloudpickle


why we use local import? And is this a requirements of TRTLLM install? if its already a requirement no need local try/catch I think. If its optional requirement, lets print a better logging for user to install it when import error?

Upaded the error message.

JunyiXu-nv · 2026-02-12T13:12:18Z

/bot run

JunyiXu-nv · 2026-02-12T13:12:37Z

/bot run

tensorrt-cicd · 2026-02-12T13:19:28Z

PR_Github #35783 [ run ] triggered by Bot. Commit: f4469af

tensorrt-cicd · 2026-02-12T13:19:32Z

PR_Github #35784 [ ] completed with state ABORTED. Commit: f4469af

tensorrt-cicd · 2026-02-12T14:04:56Z

PR_Github #35783 [ run ] completed with state SUCCESS. Commit: f4469af
/LLM/release-1.2/L0_MergeRequest_PR pipeline #351 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

JunyiXu-nv · 2026-02-13T03:33:31Z

/bot run

tensorrt-cicd · 2026-02-13T03:39:43Z

PR_Github #35866 [ run ] triggered by Bot. Commit: cda3a3a

…ssue In multi-node scenarios, AutoTokenizer.from_pretrained with trust_remote_code=True may create dynamic Python modules cached at $HOME/.cache/huggingface/modules/ transformers_modules/. These modules are node-local and won't exist on other nodes. Standard pickle serializes classes by reference, so deserializing a tokenizer that uses such a dynamic class fails on other nodes. To solve this, we use cloudpickle (if available) to serialize the inner tokenizer by value, embedding the class definition in the serialized bytes. This follows vLLM PR NVIDIA#6751. Also adds nosec comments to suppress Bandit security warnings for pickle import (B403) and pickle.loads (B301). These are safe because the serialized data comes from our own cloudpickle.dumps() call, not from untrusted external sources. See: vllm-project/vllm#6751 Signed-off-by: Junyi Xu <[email protected]>

tensorrt-cicd · 2026-02-13T06:10:28Z

PR_Github #35866 [ run ] completed with state SUCCESS. Commit: cda3a3a
/LLM/release-1.2/L0_MergeRequest_PR pipeline #363 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

JunyiXu-nv requested a review from a team as a code owner February 9, 2026 09:25

litaotju reviewed Feb 12, 2026

View reviewed changes

litaotju approved these changes Feb 12, 2026

View reviewed changes

JunyiXu-nv force-pushed the dev-junyi-fix-remote-code-issue branch from 224aea4 to f4469af Compare February 12, 2026 13:12

JunyiXu-nv force-pushed the dev-junyi-fix-remote-code-issue branch from f4469af to cda3a3a Compare February 13, 2026 03:15

JunyiXu-nv force-pushed the dev-junyi-fix-remote-code-issue branch from cda3a3a to 0c21fc3 Compare February 13, 2026 04:05

[https://nvbugs/5823783][fix] Fix multi-node trust_remote_code hang i… #11383

Are you sure you want to change the base?

[https://nvbugs/5823783][fix] Fix multi-node trust_remote_code hang i… #11383

Uh oh!

Conversation

JunyiXu-nv commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Description

Test Coverage

PR Checklist

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

JunyiXu-nv commented Feb 9, 2026

Uh oh!

coderabbitai bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

tensorrt-cicd commented Feb 9, 2026

Uh oh!

tensorrt-cicd commented Feb 9, 2026

Uh oh!

JunyiXu-nv commented Feb 9, 2026

Uh oh!

tensorrt-cicd commented Feb 9, 2026

Uh oh!

tensorrt-cicd commented Feb 9, 2026

Uh oh!

litaotju Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

JunyiXu-nv Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

JunyiXu-nv commented Feb 12, 2026

Uh oh!

JunyiXu-nv commented Feb 12, 2026

Uh oh!

tensorrt-cicd commented Feb 12, 2026

Uh oh!

tensorrt-cicd commented Feb 12, 2026

Uh oh!

tensorrt-cicd commented Feb 12, 2026

Uh oh!

JunyiXu-nv commented Feb 13, 2026

Uh oh!

tensorrt-cicd commented Feb 13, 2026

Uh oh!

tensorrt-cicd commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JunyiXu-nv commented Feb 9, 2026 •

edited

Loading

coderabbitai bot commented Feb 9, 2026 •

edited

Loading