[Feat] add keda support

[eladmotola]

keda support to the chart was added
its the defacto standart of pods autoscaling
I want to prevent the manual approach and add integration to it
FIX #xxxx (link existing issues this PR will resolve)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Detailed Checklist (Click to Expand)

Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.

[gemini-code-assist]

Summary of Changes

Hello @eladmotola, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces native KEDA autoscaling capabilities to the vLLM Production Stack Helm chart. By integrating KEDA directly, users can now configure and manage dynamic scaling of their vLLM deployments declaratively through values.yaml, moving away from manual ScaledObject creation. This enhancement simplifies the operational overhead of maintaining optimal resource utilization and responsiveness for vLLM services on Kubernetes.

Highlights

• KEDA Integration: KEDA (Kubernetes Event-driven Autoscaling) support has been integrated directly into the vLLM Production Stack Helm chart, enabling declarative autoscaling for vLLM deployments.
• Automated ScaledObject Creation: A new Helm template (helm/templates/scaledobject-vllm.yaml) automates the creation of KEDA ScaledObject resources based on Helm chart values, simplifying deployment and management.
• Enhanced Autoscaling Documentation: The autoscaling-keda.rst tutorial has been updated to reflect the new Helm-native configuration, including detailed steps for setup, verification, and advanced KEDA features like scale-to-zero and custom HPA behavior.
• Comprehensive Configuration Options: The Helm chart now exposes extensive KEDA configuration options via values.yaml, allowing users to customize scaling behavior, triggers (e.g., Prometheus queue length), and fallback mechanisms.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces KEDA support for autoscaling vLLM deployments, which is a great feature to have integrated into the Helm chart. The implementation is well-done, with a new ScaledObject template that includes a sensible default trigger. The documentation in both the README.md and values.yaml is comprehensive and clearly explains the new configuration options.

My main feedback is to improve the Prometheus query examples in the documentation and values.yaml. The current examples use queries that are not model-specific, which could lead to incorrect scaling behavior in the common multi-model deployment scenario. Making these examples more specific will provide better guidance to users.

[gemini-code-assist]

The Prometheus query in the example values.yaml is not model-specific. In a multi-model deployment, this would cause KEDA to scale based on the aggregate queue length of all models, which is likely not the desired behavior. The query should be filtered by the model name.

This also applies to the Scale-to-Zero example, where both the queue-based and traffic-based triggers should be model-specific to work correctly. For the keepalive trigger, a query like sum(increase(vllm:num_incoming_requests_total{model="llama3"}[1m])) with a threshold of 1 would be more appropriate to prevent scaling to zero as long as there is traffic to the specific model.

Suggested change

	query: vllm:num_requests_waiting
	query: 'vllm:num_requests_waiting{model="llama3"}'

[gemini-code-assist]

The example Prometheus query here is not model-specific. This could be misleading for users and cause incorrect scaling behavior in multi-model deployments. The query should be filtered by the model name to scale each model independently. The example model is mistral, so the query should filter for model="mistral".

#           query: 'vllm:num_requests_waiting{model="mistral"}'

[zerofishnoodles]

@HanFa Could you take a look at this?

[HanFa]

Thanks for the PR @eladmotola! Could you please add some unit tests under ./helm/tests to validate your stack change?

[eladmotola]

Hey @HanFa
Waiting for you review :D

[HanFa]

lgtm

[HanFa]

@zerofishnoodles mind cross checking this again? otherwise I think we are good to merge?

[zerofishnoodles]

LGTM too! Thanks for the contribution.

[zerofishnoodles]

@eladmotola Hi, could you fix the pre-commit issue?