c9s starts VMs outside of Kubernetes

[groundsada]

Hi all,

We’re trying to deploy c9s in our dual-stack, multi-tenant Kubernetes (k8s) cluster. The concern we’re running into is that it starts VMs outside of the Kubernetes control plane by mounting /dev/kvm and /dev/tun, which means those VMs run on the host outside Kubernetes’ own scheduling and resource tracking. This effectively bypasses Kubernetes’ resource accounting (CPU/Memory) and can conflict with tenant workloads on those nodes.

Two points we’d appreciate clarification on:

Does the project have plans to support KubeVirt?

Without such support, in our environment the nodes running c9s will see CPU/Memory resources consumed by out-of-band VMs, which Kubernetes cannot account for properly. Has anyone run into similar issues or have patterns to address this?

Thanks!

[carlmontanari]

Does the project have plans to support KubeVirt?

no

Without such support, in our environment the nodes running c9s will see CPU/Memory resources consumed by out-of-band VMs, which Kubernetes cannot account for properly. Has anyone run into similar issues or have patterns to address this?

afaik the cgroups stuff on the pod should still allow you to set cpu/mem limits that would apply to the stuff in the pod, incl the docker daemon and container and/or container w/ a vm inside of it in the pod. afaik kubevirt would be no different in that its just qemu in a pod anyway.

[hellt]

Hi @groundsada
no immediate plans to solve for this, but if you have time could you share if you know how kubevirt solves this?
The /dev/kvm is still being used, I believe, as this is table stakes for virtualization, but I presume they have hooks to make it accountable in k8s scheduler.
I briefly looked at their device driver, apparently this is where things are happening.

Also if you can show how is the resource issue renders itself it would be interesting to collect more feedback around it. From the scheduling perspective it "works", as the requests will be honored when we schedule the pod that runs DinD

[carlmontanari]

hmm roman smarter than me (as per usual). guess mounting /dev/kvm prolly escapes the pods limits huh

[groundsada]

Hi @carlmontanari and @hellt
As an admin, I can mount /dev/kvm and run c9s and pin CPUs and do what KubeVirt does directly
But in a multi-tenant cluster with KubeVirt, giving users the ability to mound /dev/kvm and run with privilege dishonors multi-tenancy and causes a security issue

Likewise, I can run DinD but I can't let users run DinD safely in a mutli-tenant env with thousands of users, this is where KubeVirt as basically a wrapper solves it for us

[groundsada]

That said, I'm happy to run tests

looping @dimm0 and @marcosfsch in to the discussion

[carlmontanari]

I dont think I know enough about kubevirt to really have anything smart to say :) here are general thoughts I have though:

• we require privileged pods in like... 99% of cases, so every time discussion about security related things comes up w/ c9s my reaction is: "so what, you're already doing the worst thing"
• never say never, but I pretty much see zero possibility that I would make c9s require kubevirt. in fact c9s exists because I thought requiring multus in kne was crazy

for sure interested in the discussion and am very open to reviewing/merging patches that keep pushing c9s ahead of course, but I think just generally this feels (again, im v happy to be proven wrong!) like a path that im not too optimistic about mostly for above reasons

im sure roman will have some feels/thoughts as well!