[https://nvbugs/6007967][fix] fix disagg pp hang issue

[bo-nv]

Summary by CodeRabbit

• Bug Fixes
  • Improved inter-process communication synchronization during distributed batch processing to enhance reliability and prevent potential communication delays in multi-node tensor parallelism inference deployments.
• Tests
  • Removed test waivers for disaggregated serving scenarios, confirming improved system stability and successful verification of tensor parallelism functionality across various parameter configurations.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d5843d3d-3334-4c11-8041-92ab4ba0038f

📥 Commits

Reviewing files that changed from the base of the PR and between 2b5c434 and 00c7fdb.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

---

📝 Walkthrough

Walkthrough

This pull request adds conditional MPI send flushing logic to the sample state broadcasting loop in the PyExecutor, forcing an MPI progress-driving wait when the executed batch queue is empty. Additionally, three test waivers for disaggregated serving tests are removed from the waiver list.

Changes

Cohort / File(s)	Summary
MPI Synchronization tensorrt_llm/_torch/pyexecutor/py_executor.py	Adds conditional MPI send handle waiting in _broadcast_sample_state_loop: when executed_batch_queue is empty, calls wait_on_pp_send_handles(...) on pending sends for the current microbatch before blocking on queue retrieval.
Test Waivers tests/integration/test_lists/waives.txt	Removes three test waivers for accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric parameterizations related to GSM8K/MMLU with specific ctx_pp and gen_tp configurations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is empty except for the template structure; required sections like Description and Test Coverage are not filled out with substantive information.	Add a detailed description explaining the disagg pp hang issue, the root cause, and how the fix works. Include test coverage details in the Test Coverage section.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the issue (NVBugs 6007967) and describes the fix (disagg pp hang issue), directly related to the code changes in py_executor.py.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[bo-nv]

/bot run

[tensorrt-cicd]

PR_Github #40286 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

[tensorrt-cicd]

PR_Github #40286 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31399 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[bo-nv]

/bot run

[tensorrt-cicd]

PR_Github #40335 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

[tensorrt-cicd]

PR_Github #40335 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31441 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[bo-nv]

/bot run

[tensorrt-cicd]

PR_Github #40365 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

[tensorrt-cicd]

PR_Github #40365 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31467 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[bo-nv]

/bot run

[tensorrt-cicd]

PR_Github #40397 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

[tensorrt-cicd]

PR_Github #40397 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31492 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[bo-nv]

/bot run

[tensorrt-cicd]

PR_Github #40427 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

[tensorrt-cicd]

PR_Github #40427 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31520 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation