kubernetes example

[jakexcosme]

Note: This issue was copied from ggml-org#6546

Original Author: @phymbert
Original Issue Number: ggml-org#6546
Created: 2024-04-08T16:31:37Z

---

Motivation

Kubernetes is widely used in the industry to deploy product and application at scale.

It can be useful for the community to have a llama.cpp helm chart for the server.

I have started several weeks ago, I will continue when I have more time, meanwhile any help is welcomed:

https://github.com/phymbert/llama.cpp/tree/example/kubernetes/examples/kubernetes

References

• server-cuda closes connection while still processing tasks ggml-org/llama.cpp#6545