Container metrics (cpu, memory)

[fopina]

Question

I'm migrating from fluentbit and I was happy to see better maintainers in Vector yet as soon as I start migrating I bump into something missing: container metrics (coming from fluentbit https://docs.fluentbit.io/manual/data-pipeline/inputs/docker-metrics)

I would expect, even if not having an official source for this, to be something widely discussed and documented, how to achieve it via other sources and transforms but, to my surprise, I haven't been able to find anything concrete...

It seems there was at some point a source for this and then dropped (couldn't find why or what the proposed alternative was).
It would seem I could use host metrics cgroups but then I need, somehow, to extract the POD id from the cgroup name and enrich it with k8s data, so I can properly query by pod name. ChatGPT made up a "k8s_metadata" transform that does not exist and I could not find anything similar.

I'm down to thinking I need to run some custom prom exporter that just queries k8s metrics API but I'd like to know if that's realy the only way (an extra pod over something built in vector)

Thanks

Vector Config

No response

Vector Logs

No response

[thomasqueirozb]

Hi @fopina, currently we don't have a anything like the docker-metrics source. This is tracked by #5904. You're welcome to 👍 the issue and also continue the discussion there. See also the latest update by another contributor about the status regarding this issue.

As for right now what you can do is use a combination of the host_metrics source, enrichment_tables (if you are pulling data from elsewhere and want to enrich your events), and additionally, as a last resort to get complex functionality (like file reading) inside your transforms, you can use the lua transform. Hope this helps!

[fopina]

Thanks for pointing those out @thomasqueirozb

I actually ended up going with prometheus_scrape as cadvisor is baked into k3s (which I'm using)

sources:
  pod_metrics:
      type: prometheus_scrape
      endpoints:
        - https://kubernetes.default.svc/api/v1/nodes/$VECTOR_SELF_NODE_NAME/proxy/metrics/cadvisor
      auth:
        strategy: bearer
        token: "$KUBE_TOKEN"
      tls:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      scrape_interval_secs: 15

Just had to extend the default service account in the helm chat with

rbac:
  extraRules:
    - apiGroups: [""]
      resources: ["nodes/proxy"]
      verbs: ["get"]

and a bit of command: mangling to get the token from a file into an env var before vector starts.

I hadn't seen enrichment_tables but it sounds rather tricky/unstable to call k3s/k8s API (in a cache-miss) from the pipeline to get pod info.

I've reacted to the issue you mentioned, looking forward to see any contributions there, thanks!

PS: not sure about process here, if I should close the discussion or it's left open for other people to comment

[thomasqueirozb]

I hadn't seen enrichment_tables but it sounds rather tricky/unstable to call k3s/k8s API (in a cache-miss) from the pipeline to get pod info.

it's definitely hacky in this scenario but it can be very useful if the data is available ahead of time. A lot of problems can be solved with it but I'm happy you found a better solution!

PS: not sure about process here, if I should close the discussion or it's left open for other people to comment

I'll mark an answer, that's usually the best way for everyone to see if it is effectively resolved. Also, I just tested it out and it looks like it's possible for people to comment on already closed discussions. Up to you if you want to leave open or close :)