feat(helm): Use emptyDir as pvcStorage

[Jimmy-Newtron]

I am using the vLLM stack to deploy engine servers.
I am hosted on a cloud provider that does not support the RunAI streamer to load LLM models from S3 object storage.

My goal is to temporarily bypass the S3 issue by adding an initContainer AWS CLI that download the model from S3 (faster than hugging face) and use the RunAI streamer to load the model from the local POD storage.

[gemini-code-assist]

Summary of Changes

Hello @Jimmy-Newtron, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the capability to use Kubernetes emptyDir volumes for model storage within vLLM engine deployments managed by Helm. This change provides a flexible alternative for scenarios where persistent volume claims are not suitable or available, specifically addressing a limitation with direct S3 model loading on certain cloud providers by enabling models to be downloaded to ephemeral pod-local storage via an initContainer.

Highlights

• Flexible Storage Option: I've added support for emptyDir as a pvcStorage option in the vLLM deployment Helm chart, allowing users to choose ephemeral, pod-local storage for models. This is particularly useful for temporary model downloads via an initContainer.
• Conditional Volume Provisioning: The deployment template (helm/templates/deployment-vllm-multi.yaml) now intelligently selects between a persistentVolumeClaim and an emptyDir volume based on whether pvcStorage is explicitly set to emptyDir.
• Preventing Redundant PVCs: I've updated the PVC creation logic (helm/templates/pvc.yaml) to ensure that PersistentVolumeClaims are only generated when pvcStorage is not set to emptyDir, preventing unnecessary resource provisioning and potential conflicts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the ability to use an emptyDir volume for model storage by checking for the string "emptyDir" in the pvcStorage value. I recommend setting a sizeLimit for the emptyDir to prevent node instability and refactoring the nested conditional in pvc.yaml to improve code clarity.

[Jimmy-Newtron]

Hello @YuhanLiu11 let me know your code review comments.
Thanks in advance

[YuhanLiu11]

Hey @Jimmy-Newtron if we wanna load model from S3, why don't we directly use initContainer to download s3 model to PVC and then directly load from the PVC?

[Jimmy-Newtron]

@YuhanLiu11 my current cloud provider does not allow me to use ReadWriteMany PVC for free on the kubernetes cluster I own.

Since I have a lot of spare workers disk, at least 80% free, I wanted to bypass other complex solutions.

Moreover I tried to use RunAi streamer to directly load the model from S3 but I failed.
run-ai/runai-model-streamer#69

[YuhanLiu11]

@YuhanLiu11 my current cloud provider does not allow me to use ReadWriteMany PVC for free on the kubernetes cluster I own.

Since I have a lot of spare workers disk, at least 80% free, I wanted to bypass other complex solutions.

Moreover I tried to use RunAi streamer to directly load the model from S3 but I failed. run-ai/runai-model-streamer#69

I see, this makes sense then.