Skip to content

Conversation

zetxqx
Copy link
Contributor

@zetxqx zetxqx commented Oct 18, 2025

What type of PR is this?

/kind feature

What this PR does / why we need it:
This change introduces the ability to configure the numGPUBlocks for the approximate prefix cache.

Key changes:

  • The indexer now considers numOfGPUBlocks from the server's metrics when creating a new LRU cache. This allows the cache size to be dynamically adjusted based on the server's capacity.
  • If numOfGPUBlocks is not available, a default LRU size is used.
  • The Add method in the indexer now accepts a Server struct, which includes both the ServerID and numOfGPUBlocks.

Which issue(s) this PR fixes:

Fixes partially #1304
Fixes #1512

Does this PR introduce a user-facing change?:

feat: configure LRUCacheSize using the numGPUBlocks for approximate prefix cache scorer

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 18, 2025
Copy link

netlify bot commented Oct 18, 2025

Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit 920ac60
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68f2fcd7be892c0008e5c1c8
😎 Deploy Preview https://deploy-preview-1748--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 18, 2025
@k8s-ci-robot k8s-ci-robot requested a review from elevran October 18, 2025 02:35
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zetxqx
Once this PR has been reviewed and has the lgtm label, please assign kfswain for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 18, 2025
@zetxqx
Copy link
Contributor Author

zetxqx commented Oct 18, 2025

/assign @liu-cong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-configure prefix-cache-scorer parameters from engine metrics

3 participants