Simplifying Llama Stack Deployment with Kubernetes Operator

[VaishnaviHire]

Hello everyone! I, alongwith @leseb and @rhuss would like to initiate a discussion about enhancing Llama Stack’s deployment capabilities by introducing a Kubernetes Operator.

Why the operator?
An Operator can provide several advantages for Llama-Stack server deployments -

• simplified deployments: through declarative configurations(CRD)
• automated updates : when new versions are released
• self-healing capabilities: recovering a failing instance
• dynamic scaling

making Llama Stack easier to run in a Kubernetes environment.

How it fits to Llama Stack?
The Operator could handle:

• Provisioning required resources
• Pulling and configuring the appropriate Llama Stack version and distribution
• Monitoring server status and performing automated recovery
• Scaling resources based on defined parameters

Repository Structure
To keep things modular, we propose hosting this operator in a separate GitHub repository(e.g llama-stack-k8s-operator). A separate Github repository ensures separation of concerns between the Llama Stack server and deployment strategies. It will also provide a dedicated space for Kubernetes-specific issues and contributions.

We are seeking community's feedback on leveraging Kubernetes operators for efficient deployment and management of Llama Stack server instances. Based on this feedback, we would like to share details of our prototype, including the Custom Resource Definition (CRD) and controller implementation, in subsequent discussions.

[leseb]

In addition to the Operator/Controller code, this will be a suitable location for hosting artifact deployments (Kustomize, Helm, bare YAML, etc.).

[rhuss]

IMO, it's a good idea to have all K8s relevant stuff in a single repo (so maybe calling the repo then something like llama-stack-kubernetes); we could always fan out to more specialized repos if needed later. A separate repo also allows for different release lifecycles and better parallelism in the development process, as additional committers with specialized skills can take over more responsibilities.

[VaishnaviHire]

Hi @raghotham here is the repo you requested for the k8s operator https://github.com/opendatahub-io/llama-stack-k8s-operator/tree/main. It would be great to include this under https://github.com/meta-llama

[VaishnaviHire]

Hi @raghotham, following up from the community meeting request. Did you get a chance to look at the operator repo and if we could add it under https://github.com/meta-llama ?

[franciscojavierarceo]

Can you make this an issue?

[leseb]

Done.

[leseb]

Used the magic button #1814

[leseb]

Closing, we are tracking the effort in #1814, thanks!