-
Notifications
You must be signed in to change notification settings - Fork 600
Precision synchronization issue fix #4429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Precision synchronization issue fix #4429
Conversation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request aims to fix a precision synchronization issue. My review focuses on improving code efficiency and readability. I've identified a duplicated loop in mooncake_engine.py that impacts performance and suggested a refactoring. Additionally, I've pointed out a small simplification in mooncake_store_connector_v1.py to improve code clarity.
| skip_save = False | ||
| if num_computed_token >= len(request.prompt_token_ids): | ||
| skip_save = True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This block of code can be simplified to a single line for better readability and conciseness, which is a standard Python practice.
| skip_save = False | |
| if num_computed_token >= len(request.prompt_token_ids): | |
| skip_save = True | |
| skip_save = num_computed_token >= len(request.prompt_token_ids) |
| if save_spec is None or not save_spec.can_save: | ||
| continue | ||
|
|
||
| torch.npu.synchronize() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You probably need to delete torch.npu.current_stream().synchronize() from the https://github.com/vllm-project/vllm-ascend/blob/main/vllm_ascend/distributed/mooncake/kv_transfer.py file, right? Also, is there any difference between torch.npu.current_stream().synchronize() and torch.npu.synchronize()?
What this PR does / why we need it?
Does this PR introduce any user-facing change?
How was this patch tested?