Export CPU consumption of vhost-net (VHOST) threads for domains

[insatomcat]

Hi,

First of all, thank you for maintaining the libvirt Prometheus exporter — it’s been very useful.

I’d like to ask whether you would consider exporting the CPU consumption of the vhost-net (VHOST) threads associated with domains when using virtio networking with vhost enabled.

Currently, the exporter exposes vCPU metrics, but the CPU usage of vhost threads is not included, since these threads are separate from the vCPU threads. In setups with high network throughput, this can represent a significant amount of CPU time.

As a result, we observe a noticeable gap between the total CPU consumption of the QEMU process, and the sum of the CPU consumption of the associated vCPUs

That gap can largely be explained by vhost-net threads, but today it’s invisible from the exported metrics, which makes CPU attribution and capacity analysis harder.

For context, vhost threads are per-domain (or per-queue) kernel/user-space threads created by QEMU/libvirt to accelerate virtio-net I/O, and their CPU usage can be non-trivial in network-heavy workloads.

I have a custom exporter written in Python that collects exactly this information (CPU usage of vhost threads per domain), and I’d be happy to share the code or explain the approach if that’s useful.

However, since the libvirt exporter is written in Go and I don’t have Go experience, I unfortunately can’t offer help to port or implement this myself.

Would you be open to considering this as a feature, or discussing possible approaches that would fit well with the exporter’s design?

Thanks again for your work, and happy to provide more details if needed.

[frittentheke]

@insatomcat great to hear the exporter is in use and useful!
Certainly we would add more metrics that are helpful via a PR would be most appreciated.

I guess you are referring to https://github.com/insatomcat/insatomcat-exporter/blob/cd2d6170746cef8aac83221955537b650ff742e8/insatomcat_exporter.py#L50-L52 when talking about vhost metrics?

If available, do you mind pointing to the Libvirt API that exposes these kind of metrics? https://libvirt.org/html/index.html
I am also looking at schematics like in https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net to figure out which components / threads we are talking about here and how they are even "part" of libvirt.

[insatomcat]

@frittentheke thanks for the quick reply.

Indeed this is the metric I'm referring too. However I didn't find that libvirt api was exposing it, I had to compute it myself with the psutil python module.
I get the value with "thr_cpu_s = float(thr.user_time + thr.system_time)" and I filter for the threads that contain "vhost" in their name.
I don't know if they are "part of libvirt" but when you use a virtio network interface in libvirt (<model type='virtio' />, which seems to be the default in modern libvirt) the threads are there, and they belong to the domain qemu process, so I guess the libvirt-exporter is the tool you would expect to expose the metric.