Kubernetes support

[alexevladgabriel]

This thread is intended to discuss benefits & drawbacks of having it implemented.

Are there specific aspects of Kubernetes functionality you believe would be the most beneficial for the project to support?
What are the pain points or gaps that tighter integration with Kubernetes could solve for your use cases?

[PseudoResonance]

One of the things I've been thinking about is logging. I use Kubernetes, and all my applications log to standard logs, so in Rancher, Lens, kubectl, whatever I can easily check the logs. Additionally, I have Vector collect and push those logs to a log aggregator, so that I have a nice web UI with dropdowns to choose what service I want to view logs for, etc.

With the current file logging, it can still be done, but I would need to setup a dedicated Vector instance to pull from the Pelican PV, do some custom labeling to fit with the other apps, and then aggregate it. I personally believe it'd make more sense if Pelican was built with Docker in mind, not necessarily Kubernetes, but if it's built to follow Docker standards, then it should be easy to integrate into a K8s solution. By this, I am specifically referring to again standard logging instead of files, and additionally, a more segmented approach, instead of a single monolithic container helps, as each process' logs will be divided up and labeled by the container's name automatically in K8s and Docker.

I feel like these 2 aspects are the things that most greatly affect how suitable Pelican is for running on Kubernetes, and I've been wondering why the current approach is being taken instead?

[parkervcp]

We don't log to a file. We just use dockers standard logging feature which outputs to a file.

The log config hasn't been used since 0.7 and is a left over from that.

[PseudoResonance]

It's not a direct reply to my comment, but it sounds like it might be in response to it?

Unless it's changed in the past 2 weeks since I was working on #932, Laravel logs are not output to standard logging, but are logged in storage/logs. Additionally, Supervisor logs to /var/www/html/storage/logs/supervisord/supervisord.log, which is also in that log folder. The Dockerfile is setup to have a volume mounted there to collect all the log files in one place. When I would run the container in testing, I would get effectively 0 useful logs and I had to check the logging folder for everything. That's what I am referring to.

I'm so sorry if I misunderstood something though and this wasn't a reply to my question or anything!

[parkervcp]

I was discussing the game server logging and not the panels.

[PseudoResonance]

I was discussing the game server logging and not the panels.

Ahh, I'm sorry, thank you for clarifying it for me!

[KittyKatt]

Personally, the benefit for Kubernetes comes from the scalability and resiliency. The architecture that I'd like to see (take note that this is very idealized and extremely high level without consideration for the work involved to get there) would look something like this:

• A deployment for the panel itself. This could 1 to N replicas. In my case, I'd like 3-5 replicas spaced across my worker nodes to ensure the panel is always available even through node maintenance/downtime.
• An operator that would actually orchestrate the game server deployments themselves. This would fill in a lot of what Wings currently does. Game servers could have potentially one or many replicas, depending on resources and game server type. The option to have a PVC attached to the pod hosting the game server would at least allow me resiliency when a node goes down because I'm currently using NFS-attached storage for my main PV StorageClass in my cluster. If a node were to go down for any reason, the game server pod could just pop back up on any worker node that has access to that PV.
• A plus would be reading Kubernetes node annotations/labels to determine node size/resources. This could be a dropdown selection when creating a server so if I had something particularly resource hungry, I could choose a node labelled "big". If I had a small server that probably won't take up much resources, I'd choose one of the nodes labelled "small".

This is most of what I'm looking for out of Kubernetes support in Pelican. Nearly all of my infrastructure is on my Kubernetes cluster at this point, as well, so not having to have a whole separate box/VM just for Pelican. No project (at least none that are active) that I've found currently integrates both the panel and operator functionality together to give a really good, seamless Kubernetes solution.

Hopefully that helps to outline at least one benefit to tighter integration with k8s.

[richard-callis]

The main benefit from Kubernetes that I want to take advantage is distributing the workload across the cluster, versus manually creating VMs as nodes and assigning multiple servers to one node. Plus the added benefit from longhorn managing the volumes across the cluster, it works with my already established rsync backup strategy. Claude Code has been a godsend, it has helped me transition from a traditional docker setup strategy on VMs to a distributed workload on 6 Raspberry PIs, and 2 AMD machines.

Right now the only viable path to running Wings on K8s is Docker-in-Docker (DinD) — a DinD sidecar provides a private Docker daemon, and Wings connects to it via tcp://localhost:2375. This works, but it comes with significant costs:

• Security: Both the DinD sidecar and Wings itself must run with privileged: true and runAsUser: 0. That's a full container escape surface on every game server pod. For home labs this is acceptable; for anyone running shared or multi-tenant infrastructure, it's a blocker.
• Resource visibility: Game containers launched inside DinD are completely invisible to Kubernetes. K8s cannot enforce resource limits, won't kill an OOM game process, and has no way to surface metrics. The node can be silently starved without the scheduler knowing.
• Cold start penalty: Docker image layers live in an emptyDir volume — ephemeral per pod. When Kubernetes reschedules a Wings pod to a different node (routine maintenance, node failure, etc.), DinD must re-pull all game container images from scratch. For large game servers this can mean 30–60+ minute downtime windows.
• Update downtime: Because game data lives on a ReadWriteOnce PVC, rolling updates are impossible — strategy: Recreate is
  required, meaning every Wings update drops players.

Instead of Wings managing game containers via Docker, it would speak the Kubernetes API directly — spawning game servers as Pods (or Jobs) in a namespace, with:

• No privileged: true required — game Pods get standard K8s security contexts
• Full resource enforcement — CPU/memory limits on game containers visible to the scheduler
• Native image caching — K8s node image cache survives pod rescheduling; no re-pull on restart
• Proper lifecycle management — K8s handles restarts, health probes, graceful termination
• Metrics — game Pod resource usage flows naturally into Prometheus/Grafana
• Rolling updates — Wings daemon updates without touching game Pods at all

Below is a full manifest set if useful — currently running 7 Wings instances (ARK SA, Satisfactory, Valheim, Enshrouded, 7D2D, RoR2, Moria) all on K8s with this pattern. It's stable, but the DinD privileged requirement and cold-start problem are the two things that would most benefit from a native solution.

manifests.txt