Skip to content

fix multi-GPU validation to include pipeline parallelism#116

Merged
yinggeh merged 3 commits intotriton-inference-server:mainfrom
nightflight-dk:fix/pipeline-parallel-validation
Jan 9, 2026
Merged

fix multi-GPU validation to include pipeline parallelism#116
yinggeh merged 3 commits intotriton-inference-server:mainfrom
nightflight-dk:fix/pipeline-parallel-validation

Conversation

@nightflight-dk
Copy link
Copy Markdown
Contributor

This PR updates the validation logic in TritonPythonModel to correctly identify multi-GPU configurations that use Pipeline Parallelism (PP).

Previously, the check only considered Tensor Parallelism (TP) size (tensor_parallel_size > 1) when verifying if a model configured with KIND_GPU (intended for single-GPU) was attempting to run in a multi-GPU setup. This meant that models using Pipeline Parallelism (where pipeline_parallel_size > 1) could potentially bypass this check if tensor_parallel_size was 1.

Changes:

Retrieve pipeline_parallel_size from vllm_engine_config (defaulting to 1).
Update the validation condition to check if the total parallelism (tp_size * pp_size) exceeds 1.
Ensures that users are correctly prompted to use KIND_MODEL for any multi-GPU configuration, whether it involves TP, PP, or both.
Testing:

Verified that setting pipeline_parallel_size > 1 with
KIND_GPU now raises the expected ValueError.

@nightflight-dk
Copy link
Copy Markdown
Contributor Author

@yinggeh @mc-nv pls review, thanks

yinggeh
yinggeh previously approved these changes Jan 7, 2026
Copy link
Copy Markdown
Contributor

@yinggeh yinggeh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yinggeh
Copy link
Copy Markdown
Contributor

yinggeh commented Jan 7, 2026

Can you also update the copyright to 2026 in the first line?

@yinggeh yinggeh added the bug Something isn't working label Jan 7, 2026
@yinggeh yinggeh self-requested a review January 7, 2026 07:24
@nightflight-dk
Copy link
Copy Markdown
Contributor Author

@yinggeh done, ready to merge

@yinggeh
Copy link
Copy Markdown
Contributor

yinggeh commented Jan 9, 2026

@nightflight-dk You should only update copyrights of the file you edited, which is src/model.py.

@nightflight-dk
Copy link
Copy Markdown
Contributor Author

@nightflight-dk You should only update copyrights of the file you edited, which is src/model.py.
done

@yinggeh yinggeh merged commit 41cb2c6 into triton-inference-server:main Jan 9, 2026
3 checks passed
@yinggeh
Copy link
Copy Markdown
Contributor

yinggeh commented Jan 9, 2026

Thanks for contributing. This change will be included in Triton 26.01 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants