Skip to content

Conversation

@Tabrizian
Copy link
Member

@Tabrizian Tabrizian commented Dec 18, 2025

Summary by CodeRabbit

Release Notes

  • New Features

    • Added cancellation status checking capability for LLM requests, now accessible through Python bindings for better request state visibility
  • Bug Fixes

    • Fixed improper resource reuse when requests are cancelled, preventing context blocks from being reused inappropriately
    • Ensured cancelled requests are properly cleaned up from in-flight tracking to maintain accurate request state

✏️ Tip: You can customize this high-level summary in your review settings.

Description

  • Fixed a bug to request cancellation + chunked prefill. If the request was cancelled when one of the chunks was processed, it would store all the blocks (even the ones that have not been filled out) in KVCacheManager.
  • Looks like when unpinning blocks from the last block, the path can be modified by the other requests. I changed to pin/unpin all the pinned blocks instead of relying on tree traversal.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@Tabrizian Tabrizian requested a review from a team as a code owner December 18, 2025 08:14
@Tabrizian Tabrizian requested a review from Naveassaf December 18, 2025 08:14
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 18, 2025

📝 Walkthrough

Walkthrough

This PR introduces a new isFinishedDueToCancellation() method to the GenericLlmRequest class that checks whether all per-beam finish reasons are CANCELLED. The method is exposed through Python bindings (pybind and nanobind), then integrated into the executor loop to prevent block reuse and KV cache transmission for cancelled requests and to remove cancelled requests from in-flight tracking structures.

Changes

Cohort / File(s) Summary
Core method and Python bindings
cpp/include/tensorrt_llm/batch_manager/llmRequest.h, cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp, cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp
Added isFinishedDueToCancellation() method to GenericLlmRequest in C++ header; exposed as read-only is_finished_due_to_cancellation property through both pybind and nanobind Python bindings.
Executor cancellation handling
tensorrt_llm/_torch/pyexecutor/py_executor.py
Added not req.is_finished_due_to_cancellation guard condition to prevent block reuse and KV cache transmission in _executor_loop_pp, _executor_loop, _executor_loop_overlap, and _send_disagg_ctx_cache; removes cancelled requests from ctx_in_transmission_requests in _handle_canceled_requests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20–25 minutes

  • Attention areas:
    • Verify that the cancellation guard is consistently applied across all three executor loop variants (_executor_loop_pp, _executor_loop, _executor_loop_overlap) and _send_disagg_ctx_cache to ensure no edge cases are missed.
    • Confirm that removal of cancelled requests from ctx_in_transmission_requests in _handle_canceled_requests aligns with overall request cleanup and does not leave dangling references elsewhere.
    • Check that the condition logic correctly prioritizes cancellation checks and doesn't inadvertently suppress legitimate block reuse in non-cancelled scenarios.

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
Description check ⚠️ Warning PR description addresses the bug fix and solution but Test Coverage section is empty, missing specific test cases to safeguard the changes. Add specific test cases in the Test Coverage section that verify the fix for cancellation with chunked prefill and disaggregation scenarios.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main issue being addressed: fixing a bug in the interaction between request cancellation, chunked prefill, and disaggregated execution.
✨ Finishing touches
  • 📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tensorrt_llm/_torch/pyexecutor/py_executor.py (1)

2361-2387: Fix unconditional pop on ctx_in_transmission_requests in cancellation handler

_handle_canceled_requests() now unconditionally does:

self.ctx_in_transmission_requests.pop(request.py_request_id)

after a successful _try_cancel_request. This has two problems:

  1. Definite runtime bug
    For many canceled requests, request.py_request_id will never have been inserted into ctx_in_transmission_requests (e.g., non‑disagg flows, no block reuse, requests cancelled before context completes, child/generation‑only, etc.).
    Since _try_cancel_request() returns True whenever kv_cache_transceiver is None or the request is not in transmission, this pop is reached frequently and will raise KeyError, crashing the executor on cancellation.

  2. Likely resource‑cleanup gap
    For requests that are in ctx_in_transmission_requests (block‑reuse paths), popping here bypasses the existing cleanup logic in _terminate_disagg_ctx_finished_requests() that decrements the per‑request counter and eventually calls kv_cache_manager.unpin_blocks_by_id(block_id). This risks leaking pinned KV blocks or leaving reuse bookkeeping in an inconsistent state for canceled requests.

At minimum, the pop must be guarded to avoid exceptions; ideally, canceled requests that had pending reuse should also go through a cleanup path that mirrors the unpin/decrement logic used in _terminate_disagg_ctx_finished_requests() so their KV resources are released without being reused.

Suggested minimal fix: guarded pop with optional cleanup hook
@@ def _handle_canceled_requests(self):
-            is_cancelled = self._try_cancel_request(request)
-            if is_cancelled:
+            is_cancelled = self._try_cancel_request(request)
+            if is_cancelled:
                 # Mark requests as finished, then, we reuse all existing code
                 # to clean up the KV cache resources.
                 request.finish_by_reason(FinishReason.CANCELLED)
                 request.decoding_iter = request.py_decoding_iter
-                self.ctx_in_transmission_requests.pop(request.py_request_id)
+                entry = self.ctx_in_transmission_requests.pop(
+                    request.py_request_id, None)
+                # Optionally: if entry is not None and block reuse is enabled,
+                # invoke a small helper to mirror the unpin/decrement logic
+                # from _terminate_disagg_ctx_finished_requests so any pinned
+                # reuse blocks for this request are released rather than reused.
+                # (e.g., _cleanup_reuse_state_for_cancelled_request(entry))
             else:
                 still_pending_canceled_ids.append(req_id)

You can then factor the unpin/decrement logic from _terminate_disagg_ctx_finished_requests() into a shared helper and call it here when entry is not None, to ensure canceled requests with pending reuse state are also cleaned up consistently.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 33a90f2 and 108a1c0.

📒 Files selected for processing (4)
  • cpp/include/tensorrt_llm/batch_manager/llmRequest.h (1 hunks)
  • cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp (1 hunks)
  • cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp (1 hunks)
  • tensorrt_llm/_torch/pyexecutor/py_executor.py (5 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{cpp,h,cu,cuh}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.{cpp,h,cu,cuh}: Closing braces of namespaces should have a comment saying the namespace it closes: } // namespace foo
Prefer const or constexpr variables over #define whenever possible, as the latter are not visible to the compiler
A variable that is not modified after its initialization should be declared as const
For naming of constants in C++, follow the naming section conventions
Except 0 (only used in comparison for checking signness/existence/emptiness) and nullptr, true, false, all other literals should only be used for variable initialization in C++
Use the Allman indentation style in C++
Put the semicolon for an empty for or while loop in a new line in C++
The statement forming the body of a switch, while, do .. while or for statement shall be a compound statement (use brace-delimited statements) in C++
If and else should always be followed by brace-delimited statements, even if empty or a single statement in C++
C++ filenames should use camel case with first letter lowercase: thisIsASubDir and thisIsAFilename.cpp
All files involved in the compilation of a compilation target (.exe/.so) must have filenames that are case-insensitive unique in C++
All types (including class names) in C++ should use camel case with uppercase first letter: FooBarClass
Local variables, methods and namespaces in C++ should use camel case with first letter lowercase: localFooBar
Non-magic-number global variables that are non-static and not defined in anonymous namespace in C++ should use camel case prefixed by a lower case 'g': gDontUseGlobalFoos
Non-magic-number global variables that are static or defined in an anonymous namespace in C++ should use camel case prefixed by a lower case 's': sMutableStaticGlobal
Locally visible static variables in C++ should use camel case with lowercase prefix 's' as the first letter: static std::once_flag sFlag;
Public, private and protected class member variables in C++ should use camel case prefi...

Files:

  • cpp/include/tensorrt_llm/batch_manager/llmRequest.h
  • cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp
  • cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp
**/*.h

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.h: Use a preprocessor guard in C++ header files with the format TRTLLM_<FILENAME>_H derived from the filename in all caps
The preprocessor guard name in C++ must have prefix TRTLLM_ followed by the filename, all in caps. Only use the file name, not directory names
Do not use prefix with underscore in C++ preprocessor guard symbols as such symbols are reserved in C++ standard for compilers or implementation
Do not use trailing underscore in C++ preprocessor guard symbols (unlike Google C++ guideline)

Files:

  • cpp/include/tensorrt_llm/batch_manager/llmRequest.h
**/*.{cpp,h,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the year of its latest meaningful modification

Files:

  • cpp/include/tensorrt_llm/batch_manager/llmRequest.h
  • cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp
**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces. Do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used
Python files should use snake_case naming: some_file.py
Python classes should use PascalCase naming: class SomeClass
Python functions and methods should use snake_case naming: def my_awesome_function():
Python local variables should use snake_case naming: my_variable = ...
Python variable names that start with a number should be prefixed with 'k': k_99th_percentile = ...
Python global variables should use upper snake_case with prefix 'G': G_MY_GLOBAL = ...
Python constants should use upper snake_case naming: MY_CONSTANT = ...
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings in Python for classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except to the smallest set of errors possible
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible, using the else block for logic

Files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
🧠 Learnings (7)
📚 Learning: 2025-12-12T03:27:18.859Z
Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 9655
File: tensorrt_llm/_torch/pyexecutor/sampler.py:3031-3031
Timestamp: 2025-12-12T03:27:18.859Z
Learning: In tensorrt_llm/_torch/pyexecutor/sampler.py, when reviewing code that iterates through requests, ensure it does not convert excessive data into Python lists. Instead, the code should use torch.gather or indexing to gather only the data that will be used in the for loop before converting to Python lists. This minimizes data movement and improves performance.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-12-12T03:27:08.565Z
Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 9655
File: tensorrt_llm/_torch/pyexecutor/sampler.py:3031-3031
Timestamp: 2025-12-12T03:27:08.565Z
Learning: In files under tensorrt_llm/_torch/pyexecutor, avoid accessing torch.Tensor objects inside for-loops when iterating over requests. Convert batched tensors to Python lists beforehand using tensor.tolist(), and then iterate over those lists. This improves performance by reducing tensor-bound operations inside hot loops. Apply this pattern to similar code paths that process batches to access simple Python data structures (lists) inside loops.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-07-17T09:01:27.402Z
Learnt from: amitz-nv
Repo: NVIDIA/TensorRT-LLM PR: 5616
File: tensorrt_llm/executor/worker.py:375-384
Timestamp: 2025-07-17T09:01:27.402Z
Learning: In tensorrt_llm/executor/worker.py, the LoRA adapter cache optimization logic that checks `is_adapter_in_cpu_cache()` and conditionally passes None for weights/config has a known race condition issue that cannot be solved with simple error handling or verification checks. This is a known limitation that requires a more comprehensive solution.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-21T09:41:49.347Z
Learnt from: eopXD
Repo: NVIDIA/TensorRT-LLM PR: 6768
File: cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp:2010-2045
Timestamp: 2025-08-21T09:41:49.347Z
Learning: In cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp, updateSequenceCacheBlockOffsets is specifically for updating bookkeeping when blocks are added during the context phase, not for refreshing offsets after detach operations. During detach operations, GenerationRequest::removeFrontBlock handles the necessary cache block bookkeeping internally.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-14T21:04:50.248Z
Learnt from: thorjohnsen
Repo: NVIDIA/TensorRT-LLM PR: 6910
File: cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp:0-0
Timestamp: 2025-08-14T21:04:50.248Z
Learning: In KV cache onboarding logic during prefill in cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp, when calculating which blocks fall within the attention window, use getTokensPerBlock() to advance token indices rather than block->getUniqueTokens().size(), because the calculation needs to consider the post-prefill state where blocks will be filled to capacity, not their current token count.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-19T12:45:11.997Z
Learnt from: amitz-nv
Repo: NVIDIA/TensorRT-LLM PR: 7033
File: tensorrt_llm/_torch/pyexecutor/model_engine.py:0-0
Timestamp: 2025-08-19T12:45:11.997Z
Learning: In tensorrt_llm/_torch/pyexecutor/model_engine.py, DoRA (Delta Orthogonal Rank Adaptation) functionality was removed from the PyTorch flow to eliminate issues with inverted DoRA detection logic. The original is_dora condition was checking if scaling_vec_pointer == 0, which was potentially incorrect.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-28T10:22:02.288Z
Learnt from: ixlmar
Repo: NVIDIA/TensorRT-LLM PR: 7294
File: tensorrt_llm/_torch/pyexecutor/sampler.py:1191-1197
Timestamp: 2025-08-28T10:22:02.288Z
Learning: In tensorrt_llm/_torch/pyexecutor/sampler.py, the object identity comparison `softmax_req_indices is not group_req_indices_cuda` on line ~1191 is intentional and used as an optimization to determine whether to reuse an existing indexer or create a new one, based on which code path was taken during tensor assignment.

Applied to files:

  • tensorrt_llm/_torch/pyexecutor/py_executor.py
🧬 Code graph analysis (1)
cpp/include/tensorrt_llm/batch_manager/llmRequest.h (1)
cpp/include/tensorrt_llm/executor/types.h (1)
  • FinishReason (503-598)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Pre-commit Check
🔇 Additional comments (4)
cpp/include/tensorrt_llm/batch_manager/llmRequest.h (1)

1670-1674: Cancellation helper mirrors length-based check (LGTM)

isFinishedDueToCancellation() cleanly mirrors isFinishedDueToLength() by using std::all_of over mFinishReasons and checking for kCANCELLED. Naming, noexcept, and [[nodiscard]] usage are consistent with existing helpers.

cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp (1)

168-168: Expose new cancellation helper via pybind (LGTM)

The is_finished_due_to_cancellation read‑only property correctly binds to GenLlmReq::isFinishedDueToCancellation and complements is_finished_due_to_length without changing behavior elsewhere.

cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp (1)

164-164: Expose new cancellation helper via nanobind (LGTM)

The is_finished_due_to_cancellation read‑only property correctly mirrors the pybind binding and wires to GenLlmReq::isFinishedDueToCancellation as intended.

tensorrt_llm/_torch/pyexecutor/py_executor.py (1)

1070-1076: Avoid KV cache reuse / disagg ctx send for canceled context‑only requests (LGTM)

The added and not req.is_finished_due_to_cancellation checks in:

  • _executor_loop_pp (previous_batch.scheduled_ctx_reqs),
  • _executor_loop (scheduled_batch.context_requests),
  • _executor_loop_overlap (previous_batch.sample_state.scheduled_requests.context_requests), and
  • _send_disagg_ctx_cache,

correctly prevent both KV cache block reuse and disaggregated context KV transfer for context‑only requests that have finished purely due to cancellation. This matches the new C++ helper semantics and keeps the existing length‑based logic intact.

Also applies to: 1341-1347, 1570-1576, 2082-2085

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 108a1c0 to 655832d Compare December 19, 2025 19:52
@Tabrizian Tabrizian requested a review from a team as a code owner December 19, 2025 19:52
@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 655832d to fc9d685 Compare December 19, 2025 20:44
@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from fc9d685 to 4363c3e Compare December 19, 2025 21:09
@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #29177 [ run ] triggered by Bot. Commit: 4363c3e

@tensorrt-cicd
Copy link
Collaborator

PR_Github #29177 [ run ] completed with state SUCCESS. Commit: 4363c3e
/LLM/main/L0_MergeRequest_PR pipeline #22382 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

{
block->decRefCount();
if (!block->hasRefs())
if (blockId < 0 || static_cast<size_t>(blockId) >= mAllBlocksById.size())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious if there is a use case for this check or if we should warn/assert instead?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this should not fail silently.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Copy link
Collaborator

@eopXD eopXD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e2e test coverage please

store_blocks_for_reuse: bool = True):
# When store_blocks_for_reuse is False, pass None to prevent block storage
llm_request = request if store_blocks_for_reuse else None
return self.impl.remove_sequence(request.py_request_id, llm_request,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remove sequence function checks if block reuse is enabled. Line 671 can be removed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal here is different. It is to avoid storing blocks for reuse twice for context only requests (it is done already in line py_executor.py#L1574).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this logic for now. It is not critical to fix this bug.

[](tbk::BaseKVCacheManager& self, tb::LlmRequest::RequestIdType requestId, tb::LlmRequest const* llmRequest,
bool pinOnRelease)
{
if (llmRequest != nullptr)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto, remove sequenceSequence will check if the reuse toggle is on. The if-else here can be avoided.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to control whether removeSequence stores block for reuse too or not.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here determines whether we shall store it or not already.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is for the case when block reuse is already enabled but we want to control whether during the free_resources call we want to store the blocks for reuse too or not. For disaggregated serving we have already stored the blocks for reuse in (py_executor.py#L1574) so want to avoid traversing the tree twice. Would be happy to chat more if it is not clear.

{
block->decRefCount();
if (!block->hasRefs())
if (blockId < 0 || static_cast<size_t>(blockId) >= mAllBlocksById.size())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this should not fail silently.

if (pinBlocks)
{
searchRoot->incRefCount();
pinnedBlockIds.push_back(searchRoot->getBlockId());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have a block already matched and in the search tree, do we need to pin it again?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pin block is only used in disaggregated serving. The goal is to make sure the blocks that are needed for the generation server are not evicted. The cycle is:

  1. Context server stores and increases the ref count by one additional number if block reuse is enabled and it is a context-only request.
  2. The decode server starts fetching those blocks from the reuse tree.
  3. The context server decreases the ref count once the transmission has been completed.

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch 2 times, most recently from 489fe29 to 2939fb5 Compare January 1, 2026 19:30
@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30350 [ run ] triggered by Bot. Commit: 2939fb5

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30350 [ run ] completed with state SUCCESS. Commit: 2939fb5
/LLM/main/L0_MergeRequest_PR pipeline #23382 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30356 [ run ] triggered by Bot. Commit: 2939fb5

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30356 [ run ] completed with state SUCCESS. Commit: 2939fb5
/LLM/main/L0_MergeRequest_PR pipeline #23387 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30367 [ run ] triggered by Bot. Commit: 2939fb5

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 2939fb5 to 37b04f3 Compare January 2, 2026 05:09
@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30373 [ run ] triggered by Bot. Commit: 37b04f3

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30373 [ run ] completed with state SUCCESS. Commit: 37b04f3
/LLM/main/L0_MergeRequest_PR pipeline #23403 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 37b04f3 to 59c0b0f Compare January 2, 2026 16:19
@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30391 [ run ] triggered by Bot. Commit: 59c0b0f

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30391 [ run ] completed with state SUCCESS. Commit: 59c0b0f
/LLM/main/L0_MergeRequest_PR pipeline #23420 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30414 [ run ] triggered by Bot. Commit: 59c0b0f

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30414 [ run ] completed with state SUCCESS. Commit: 59c0b0f
/LLM/main/L0_MergeRequest_PR pipeline #23442 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 59c0b0f to 65821ce Compare January 3, 2026 01:36
@Tabrizian
Copy link
Member Author

/bot run --disable-fail-fast

@Tabrizian Tabrizian force-pushed the user/imant/fixCancellationRequestBug branch from 65821ce to a6074a0 Compare January 3, 2026 01:38
@tensorrt-cicd
Copy link
Collaborator

PR_Github #30425 [ run ] triggered by Bot. Commit: a6074a0

@tensorrt-cicd
Copy link
Collaborator

PR_Github #30425 [ run ] completed with state SUCCESS. Commit: a6074a0
/LLM/main/L0_MergeRequest_PR pipeline #23453 completed with status: 'FAILURE'

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants