High CPU Usage with Vector's Route Component in Large K8s Cluster

[P0lskay]

I am using Vector in a large Kubernetes cluster where a single node can run up to 400 pods, generating up to 10k logs per second. In my Vector configuration, I am utilizing the transform component with the route function to create up to 400 different streams (split by namespace). Each of these streams is then passed through the throttle component with individual parameters before being merged back into a single stream.

With this configuration, the Vector pod can consume 5-6 CPU cores, which seems excessively high. I am wondering if the route operation is inherently resource-intensive or if there are any optimization strategies I can employ to reduce CPU usage.

Questions:

Is the route function known to be CPU-intensive, especially when handling a large number of streams?
Are there any best practices or alternative configurations that could help optimize CPU usage in this scenario?
Could there be other factors in my setup contributing to high CPU usage that I should investigate?
Additional Information:

Vector version: 0.44.0
Kubernetes version: 1.30.1

Thank you for your assistance!

[P0lskay]

@jszwedko Hello! Sorry for tagging. You once helped me solve a Vector problem. May I ask you about the problem in this discussion?