[Feature] Support ARC eviction policy for NPU offloader

[ZhongsJie]

What this PR does / why we need it?

The upstream vLLM project has already added support for the ARC cache eviction policy in the CPU offloader (see vllm-project/vllm#27039). This PR adapts the same functionality for the NPU.

Does this PR introduce any user-facing change?

How was this patch tested?

• vLLM version: v0.16.0
• vLLM main: vllm-project/vllm@4034c3d

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the NPU offloader by integrating the Adaptive Replacement Cache (ARC) eviction policy, aligning its functionality with recent upstream vLLM developments for CPU offloaders. This change provides greater flexibility in managing KV cache offloading, potentially improving performance and efficiency for NPU-based systems by offering a more sophisticated cache replacement algorithm.

Highlights

• ARC Eviction Policy Support: Implemented support for the ARC (Adaptive Replacement Cache) eviction policy within the NPU offloader, providing an alternative to the existing LRU policy.
• Configurable Eviction Policy: The NPU offloader now allows specifying the eviction policy ('lru' or 'arc') via configuration, defaulting to 'lru' if not specified.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• vllm_ascend/kv_offload/npu.py
  • Imported ARCOffloadingManager for ARC eviction policy support.
  • Added an 'eviction_policy' attribute to the NPUOffloader class, configurable via extra_config.
  • Modified the get_manager method to dynamically instantiate either LRUOffloadingManager or ARCOffloadingManager based on the configured eviction policy.

Activity

• No specific activity (comments, reviews, or progress updates) was found in the provided context for this pull request.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for the ARC eviction policy for the NPU offloader. While the implementation seems correct, there are several high-severity issues that must be addressed before merging:

1. Lack of Testing: The new ARC eviction policy is introduced without any tests. It is critical to add tests to validate its functionality and prevent future regressions. You could, for example, extend tests/e2e/singlecard/test_cpu_offloading.py to cover the arc policy.

2. Incomplete Pull Request Description: The PR description is missing key information. The sections 'Does this PR introduce any user-facing change?' and 'How was this patch tested?' must be filled out. Note that adding the eviction_policy option is a user-facing change.

3. Style Guide Violation: The PR title and description do not follow the repository's style guide. As requested by the guide, here are suggestions to align with the required format:

Suggested PR Title:

[NPU][Feature] Support ARC eviction policy for NPU offloader

Suggested PR Summary:

### What this PR does / why we need it?

The upstream vLLM project has already added support for the ARC cache eviction policy in the CPU offloader. This PR adapts the same functionality for the NPU, allowing users to select 'arc' as the `eviction_policy` in `kv_connector_extra_config`.

Fixes #

### Does this PR introduce _any_ user-facing change?

Yes. This PR introduces a new `eviction_policy` option in `kv_connector_extra_config` for the NPU offloader. It can be set to `lru` (default) or `arc`.

### How was this patch tested?

<!--
CI passed with new added/existing test.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->

[ZhongsJie]

@nalinaly @MengqingCao Could you spare some time to help review it?

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.