[Misc] Getting and passing ray runtime_env to workers #22040
Conversation
ruisearch42
requested review from
simon-mo,
WoosukKwon,
youkaichao,
robertgshaw2-redhat,
mgoin,
tlrmchlsmth,
houseroad and
hmellor
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to capture the Ray runtime_env when creating an engine configuration. This is a valuable feature for distributed deployments on Ray. The implementation correctly captures the runtime_env and stores it in the ParallelConfig. However, my review found a critical issue where this captured runtime_env is not actually used to configure the Ray workers, making the feature incomplete. The workers' runtime environment will not be affected by this change as it stands.
ruisearch42
force-pushed
the
rt_env
branch
2 times, most recently
from
4502c08 to
20f243d
Compare
Signed-off-by: Rui Qiao <[email protected]>
Signed-off-by: Rui Qiao <[email protected]>
Signed-off-by: Rui Qiao <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Noam Gat <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Paul Pak <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]> Signed-off-by: Diego-Castan <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]>
…2040) Signed-off-by: Rui Qiao <[email protected]>
| # as opposed to is_in_ray_actor(). | ||
| import ray | ||
| ray_runtime_env = ray.get_runtime_context().runtime_env | ||
| logger.info("Using ray runtime env: %s", ray_runtime_env) |
There was a problem hiding this comment.
Please don't print things like envs to logs, they often can contain secrets.
There was a problem hiding this comment.
Thanks, make sense. I created a PR to address this. #26302
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
Currently vLLM does not pass down Ray runtime_env to the workers. This PR fixes it.
Fixes ray-project/ray#54866
Co-authored with @eric-higgins-ai from #21143
Test Plan
Added new unit test.
Manual test of #54866
Verified flashinfer dep is passed down and taking effect:
Test Result
Passed
(Optional) Documentation Update