Skip to content

adjust CUDA_VISIBLE_DEVICES for TP > 1 #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 11 commits into
base: heterogenous-tp
Choose a base branch
from
Open

adjust CUDA_VISIBLE_DEVICES for TP > 1 #1

wants to merge 11 commits into from

Conversation

@ptarasiewiczNV
Copy link

@ptarasiewiczNV ptarasiewiczNV commented May 16, 2025 •
edited by github-actions bot
Loading

When TP > 1 CUDA_VISIBLE_DEVICES should include all required devices. Also adjust decoder side channel ports for TP > 1.

NickLucche and others added 10 commits May 13, 2025 07:38
@NickLucche
split kv_cache along head dim
afccc5f 
fix descr indexing

change remote worker selection indexing; test ptp2-dtp4

Signed-off-by: nicklucche <[email protected]>
@NickLucche
rebase cruft
e96f832 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
address race condition
f0ba728 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
integration test
d0c44bb 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
address review: move src_xfer_side_handle init out of hotpath and docs
bde92a5 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
address review: clarity
f674fc7 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
typing
230a8b8 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
revert proxy server
8274635 
Signed-off-by: nicklucche <[email protected]>
@NickLucche
fix req_finished sync bug
abc710a 
Signed-off-by: nicklucche <[email protected]>
@ptarasiewiczNV
adjust CUDA_VISIBLE_DEVICES for TP > 1
d5d3184
@github-actions
Copy link

github-actions bot commented May 16, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ptarasiewiczNV @NickLucche