-
Notifications
You must be signed in to change notification settings - Fork 2k
[https://nvbugs/5689235][fix] Fix cancellation+chunked prefill+disagg #10111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[https://nvbugs/5689235][fix] Fix cancellation+chunked prefill+disagg #10111
Conversation
📝 WalkthroughWalkthroughThis PR introduces a new Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20–25 minutes
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tensorrt_llm/_torch/pyexecutor/py_executor.py (1)
2361-2387: Fix unconditional pop onctx_in_transmission_requestsin cancellation handler
_handle_canceled_requests()now unconditionally does:self.ctx_in_transmission_requests.pop(request.py_request_id)after a successful
_try_cancel_request. This has two problems:
Definite runtime bug
For many canceled requests,request.py_request_idwill never have been inserted intoctx_in_transmission_requests(e.g., non‑disagg flows, no block reuse, requests cancelled before context completes, child/generation‑only, etc.).
Since_try_cancel_request()returnsTruewheneverkv_cache_transceiverisNoneor the request is not in transmission, thispopis reached frequently and will raiseKeyError, crashing the executor on cancellation.Likely resource‑cleanup gap
For requests that are inctx_in_transmission_requests(block‑reuse paths), popping here bypasses the existing cleanup logic in_terminate_disagg_ctx_finished_requests()that decrements the per‑request counter and eventually callskv_cache_manager.unpin_blocks_by_id(block_id). This risks leaking pinned KV blocks or leaving reuse bookkeeping in an inconsistent state for canceled requests.At minimum, the
popmust be guarded to avoid exceptions; ideally, canceled requests that had pending reuse should also go through a cleanup path that mirrors the unpin/decrement logic used in_terminate_disagg_ctx_finished_requests()so their KV resources are released without being reused.Suggested minimal fix: guarded pop with optional cleanup hook
@@ def _handle_canceled_requests(self): - is_cancelled = self._try_cancel_request(request) - if is_cancelled: + is_cancelled = self._try_cancel_request(request) + if is_cancelled: # Mark requests as finished, then, we reuse all existing code # to clean up the KV cache resources. request.finish_by_reason(FinishReason.CANCELLED) request.decoding_iter = request.py_decoding_iter - self.ctx_in_transmission_requests.pop(request.py_request_id) + entry = self.ctx_in_transmission_requests.pop( + request.py_request_id, None) + # Optionally: if entry is not None and block reuse is enabled, + # invoke a small helper to mirror the unpin/decrement logic + # from _terminate_disagg_ctx_finished_requests so any pinned + # reuse blocks for this request are released rather than reused. + # (e.g., _cleanup_reuse_state_for_cancelled_request(entry)) else: still_pending_canceled_ids.append(req_id)You can then factor the unpin/decrement logic from
_terminate_disagg_ctx_finished_requests()into a shared helper and call it here whenentryis notNone, to ensure canceled requests with pending reuse state are also cleaned up consistently.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
cpp/include/tensorrt_llm/batch_manager/llmRequest.h(1 hunks)cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp(1 hunks)cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp(1 hunks)tensorrt_llm/_torch/pyexecutor/py_executor.py(5 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{cpp,h,cu,cuh}
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
**/*.{cpp,h,cu,cuh}: Closing braces of namespaces should have a comment saying the namespace it closes:} // namespace foo
Preferconstorconstexprvariables over#definewhenever possible, as the latter are not visible to the compiler
A variable that is not modified after its initialization should be declared asconst
For naming of constants in C++, follow the naming section conventions
Except0(only used in comparison for checking signness/existence/emptiness) andnullptr,true,false, all other literals should only be used for variable initialization in C++
Use the Allman indentation style in C++
Put the semicolon for an emptyfororwhileloop in a new line in C++
The statement forming the body of aswitch,while,do .. whileorforstatement shall be a compound statement (use brace-delimited statements) in C++
If and else should always be followed by brace-delimited statements, even if empty or a single statement in C++
C++ filenames should use camel case with first letter lowercase:thisIsASubDirandthisIsAFilename.cpp
All files involved in the compilation of a compilation target (.exe/.so) must have filenames that are case-insensitive unique in C++
All types (including class names) in C++ should use camel case with uppercase first letter:FooBarClass
Local variables, methods and namespaces in C++ should use camel case with first letter lowercase:localFooBar
Non-magic-number global variables that are non-static and not defined in anonymous namespace in C++ should use camel case prefixed by a lower case 'g':gDontUseGlobalFoos
Non-magic-number global variables that are static or defined in an anonymous namespace in C++ should use camel case prefixed by a lower case 's':sMutableStaticGlobal
Locally visible static variables in C++ should use camel case with lowercase prefix 's' as the first letter:static std::once_flag sFlag;
Public, private and protected class member variables in C++ should use camel case prefi...
Files:
cpp/include/tensorrt_llm/batch_manager/llmRequest.hcpp/tensorrt_llm/nanobind/batch_manager/bindings.cppcpp/tensorrt_llm/pybind/batch_manager/bindings.cpp
**/*.h
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
**/*.h: Use a preprocessor guard in C++ header files with the formatTRTLLM_<FILENAME>_Hderived from the filename in all caps
The preprocessor guard name in C++ must have prefixTRTLLM_followed by the filename, all in caps. Only use the file name, not directory names
Do not use prefix with underscore in C++ preprocessor guard symbols as such symbols are reserved in C++ standard for compilers or implementation
Do not use trailing underscore in C++ preprocessor guard symbols (unlike Google C++ guideline)
Files:
cpp/include/tensorrt_llm/batch_manager/llmRequest.h
**/*.{cpp,h,cu,cuh,py}
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the year of its latest meaningful modification
Files:
cpp/include/tensorrt_llm/batch_manager/llmRequest.hcpp/tensorrt_llm/nanobind/batch_manager/bindings.cpptensorrt_llm/_torch/pyexecutor/py_executor.pycpp/tensorrt_llm/pybind/batch_manager/bindings.cpp
**/*.py
📄 CodeRabbit inference engine (CODING_GUIDELINES.md)
**/*.py: Code developed for TensorRT-LLM should conform to Python 3.8+
Indent Python code with 4 spaces. Do not use tabs
Always maintain the namespace when importing in Python, even if only one class or function from a module is used
Python files should use snake_case naming:some_file.py
Python classes should use PascalCase naming:class SomeClass
Python functions and methods should use snake_case naming:def my_awesome_function():
Python local variables should use snake_case naming:my_variable = ...
Python variable names that start with a number should be prefixed with 'k':k_99th_percentile = ...
Python global variables should use upper snake_case with prefix 'G':G_MY_GLOBAL = ...
Python constants should use upper snake_case naming:MY_CONSTANT = ...
Avoid shadowing variables declared in an outer scope in Python
Initialize all externally visible members of a Python class in the constructor
For Python interfaces that may be used outside a file, prefer docstrings over comments
Python comments should be reserved for code within a function, or interfaces that are local to a file
Use Google style docstrings in Python for classes and functions, which can be parsed by Sphinx
Python attributes and variables can be documented inline with type and description
Avoid using reflection in Python when functionality can be easily achieved without reflection
When using try-except blocks in Python, limit the except to the smallest set of errors possible
When using try-except blocks in Python to handle multiple possible variable types (duck-typing), keep the body of the try as small as possible, using the else block for logic
Files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
🧠 Learnings (7)
📚 Learning: 2025-12-12T03:27:18.859Z
Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 9655
File: tensorrt_llm/_torch/pyexecutor/sampler.py:3031-3031
Timestamp: 2025-12-12T03:27:18.859Z
Learning: In tensorrt_llm/_torch/pyexecutor/sampler.py, when reviewing code that iterates through requests, ensure it does not convert excessive data into Python lists. Instead, the code should use torch.gather or indexing to gather only the data that will be used in the for loop before converting to Python lists. This minimizes data movement and improves performance.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-12-12T03:27:08.565Z
Learnt from: tongyuantongyu
Repo: NVIDIA/TensorRT-LLM PR: 9655
File: tensorrt_llm/_torch/pyexecutor/sampler.py:3031-3031
Timestamp: 2025-12-12T03:27:08.565Z
Learning: In files under tensorrt_llm/_torch/pyexecutor, avoid accessing torch.Tensor objects inside for-loops when iterating over requests. Convert batched tensors to Python lists beforehand using tensor.tolist(), and then iterate over those lists. This improves performance by reducing tensor-bound operations inside hot loops. Apply this pattern to similar code paths that process batches to access simple Python data structures (lists) inside loops.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-07-17T09:01:27.402Z
Learnt from: amitz-nv
Repo: NVIDIA/TensorRT-LLM PR: 5616
File: tensorrt_llm/executor/worker.py:375-384
Timestamp: 2025-07-17T09:01:27.402Z
Learning: In tensorrt_llm/executor/worker.py, the LoRA adapter cache optimization logic that checks `is_adapter_in_cpu_cache()` and conditionally passes None for weights/config has a known race condition issue that cannot be solved with simple error handling or verification checks. This is a known limitation that requires a more comprehensive solution.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-21T09:41:49.347Z
Learnt from: eopXD
Repo: NVIDIA/TensorRT-LLM PR: 6768
File: cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp:2010-2045
Timestamp: 2025-08-21T09:41:49.347Z
Learning: In cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp, updateSequenceCacheBlockOffsets is specifically for updating bookkeeping when blocks are added during the context phase, not for refreshing offsets after detach operations. During detach operations, GenerationRequest::removeFrontBlock handles the necessary cache block bookkeeping internally.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-14T21:04:50.248Z
Learnt from: thorjohnsen
Repo: NVIDIA/TensorRT-LLM PR: 6910
File: cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp:0-0
Timestamp: 2025-08-14T21:04:50.248Z
Learning: In KV cache onboarding logic during prefill in cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp, when calculating which blocks fall within the attention window, use getTokensPerBlock() to advance token indices rather than block->getUniqueTokens().size(), because the calculation needs to consider the post-prefill state where blocks will be filled to capacity, not their current token count.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-19T12:45:11.997Z
Learnt from: amitz-nv
Repo: NVIDIA/TensorRT-LLM PR: 7033
File: tensorrt_llm/_torch/pyexecutor/model_engine.py:0-0
Timestamp: 2025-08-19T12:45:11.997Z
Learning: In tensorrt_llm/_torch/pyexecutor/model_engine.py, DoRA (Delta Orthogonal Rank Adaptation) functionality was removed from the PyTorch flow to eliminate issues with inverted DoRA detection logic. The original is_dora condition was checking if scaling_vec_pointer == 0, which was potentially incorrect.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
📚 Learning: 2025-08-28T10:22:02.288Z
Learnt from: ixlmar
Repo: NVIDIA/TensorRT-LLM PR: 7294
File: tensorrt_llm/_torch/pyexecutor/sampler.py:1191-1197
Timestamp: 2025-08-28T10:22:02.288Z
Learning: In tensorrt_llm/_torch/pyexecutor/sampler.py, the object identity comparison `softmax_req_indices is not group_req_indices_cuda` on line ~1191 is intentional and used as an optimization to determine whether to reuse an existing indexer or create a new one, based on which code path was taken during tensor assignment.
Applied to files:
tensorrt_llm/_torch/pyexecutor/py_executor.py
🧬 Code graph analysis (1)
cpp/include/tensorrt_llm/batch_manager/llmRequest.h (1)
cpp/include/tensorrt_llm/executor/types.h (1)
FinishReason(503-598)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (4)
cpp/include/tensorrt_llm/batch_manager/llmRequest.h (1)
1670-1674: Cancellation helper mirrors length-based check (LGTM)
isFinishedDueToCancellation()cleanly mirrorsisFinishedDueToLength()by usingstd::all_ofovermFinishReasonsand checking forkCANCELLED. Naming, noexcept, and [[nodiscard]] usage are consistent with existing helpers.cpp/tensorrt_llm/pybind/batch_manager/bindings.cpp (1)
168-168: Expose new cancellation helper via pybind (LGTM)The
is_finished_due_to_cancellationread‑only property correctly binds toGenLlmReq::isFinishedDueToCancellationand complementsis_finished_due_to_lengthwithout changing behavior elsewhere.cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp (1)
164-164: Expose new cancellation helper via nanobind (LGTM)The
is_finished_due_to_cancellationread‑only property correctly mirrors the pybind binding and wires toGenLlmReq::isFinishedDueToCancellationas intended.tensorrt_llm/_torch/pyexecutor/py_executor.py (1)
1070-1076: Avoid KV cache reuse / disagg ctx send for canceled context‑only requests (LGTM)The added
and not req.is_finished_due_to_cancellationchecks in:
_executor_loop_pp(previous_batch.scheduled_ctx_reqs),_executor_loop(scheduled_batch.context_requests),_executor_loop_overlap(previous_batch.sample_state.scheduled_requests.context_requests), and_send_disagg_ctx_cache,correctly prevent both KV cache block reuse and disaggregated context KV transfer for context‑only requests that have finished purely due to cancellation. This matches the new C++ helper semantics and keeps the existing length‑based logic intact.
Also applies to: 1341-1347, 1570-1576, 2082-2085
108a1c0 to
655832d
Compare
655832d to
fc9d685
Compare
fc9d685 to
4363c3e
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #29177 [ run ] triggered by Bot. Commit: |
|
PR_Github #29177 [ run ] completed with state
|
| { | ||
| block->decRefCount(); | ||
| if (!block->hasRefs()) | ||
| if (blockId < 0 || static_cast<size_t>(blockId) >= mAllBlocksById.size()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious if there is a use case for this check or if we should warn/assert instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, this should not fail silently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated.
eopXD
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e2e test coverage please
| store_blocks_for_reuse: bool = True): | ||
| # When store_blocks_for_reuse is False, pass None to prevent block storage | ||
| llm_request = request if store_blocks_for_reuse else None | ||
| return self.impl.remove_sequence(request.py_request_id, llm_request, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The remove sequence function checks if block reuse is enabled. Line 671 can be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The goal here is different. It is to avoid storing blocks for reuse twice for context only requests (it is done already in line py_executor.py#L1574).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed this logic for now. It is not critical to fix this bug.
| [](tbk::BaseKVCacheManager& self, tb::LlmRequest::RequestIdType requestId, tb::LlmRequest const* llmRequest, | ||
| bool pinOnRelease) | ||
| { | ||
| if (llmRequest != nullptr) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto, remove sequenceSequence will check if the reuse toggle is on. The if-else here can be avoided.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to control whether removeSequence stores block for reuse too or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The logic here determines whether we shall store it or not already.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is for the case when block reuse is already enabled but we want to control whether during the free_resources call we want to store the blocks for reuse too or not. For disaggregated serving we have already stored the blocks for reuse in (py_executor.py#L1574) so want to avoid traversing the tree twice. Would be happy to chat more if it is not clear.
| { | ||
| block->decRefCount(); | ||
| if (!block->hasRefs()) | ||
| if (blockId < 0 || static_cast<size_t>(blockId) >= mAllBlocksById.size()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, this should not fail silently.
| if (pinBlocks) | ||
| { | ||
| searchRoot->incRefCount(); | ||
| pinnedBlockIds.push_back(searchRoot->getBlockId()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we have a block already matched and in the search tree, do we need to pin it again?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pin block is only used in disaggregated serving. The goal is to make sure the blocks that are needed for the generation server are not evicted. The cycle is:
- Context server stores and increases the ref count by one additional number if block reuse is enabled and it is a context-only request.
- The decode server starts fetching those blocks from the reuse tree.
- The context server decreases the ref count once the transmission has been completed.
489fe29 to
2939fb5
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #30350 [ run ] triggered by Bot. Commit: |
|
PR_Github #30350 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #30356 [ run ] triggered by Bot. Commit: |
|
PR_Github #30356 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #30367 [ run ] triggered by Bot. Commit: |
2939fb5 to
37b04f3
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #30373 [ run ] triggered by Bot. Commit: |
|
PR_Github #30373 [ run ] completed with state
|
37b04f3 to
59c0b0f
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #30391 [ run ] triggered by Bot. Commit: |
|
PR_Github #30391 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #30414 [ run ] triggered by Bot. Commit: |
|
PR_Github #30414 [ run ] completed with state
|
59c0b0f to
65821ce
Compare
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
|
/bot run --disable-fail-fast |
65821ce to
a6074a0
Compare
|
PR_Github #30425 [ run ] triggered by Bot. Commit: |
|
PR_Github #30425 [ run ] completed with state
|
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
✏️ Tip: You can customize this high-level summary in your review settings.
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.