Kubernetes CPU Usage Doesn't Match Load Test

[yoelsusanto]

Context

Hi guys, I am testing to kubernetes CPU usage to see if it matches with the load test I run.

For testing, here is the stack that I use:

• App: simple golang web server. The app receives a POST request then runs sha256 on a simple text. To simulate CPU load, this operation will be repeated as specified in the request.
• Monitoring: kube-prometheus
• Load generator: k6

For the test, I created a load pattern consisting of 1 minute of 5 virtual users then ramp up to 50 users for 1 minute and then repeat again.

Problem

I checked the CPU load on grafana to see if it matches the load pattern I have generated. However I found out that it doesn't really align with the number of user.

Here is what I think is confusing from the CPU Usage graph:

• CPU load at 00.06.00 - 00.07.00 went up but doesn't reach the same peak as those in 00.11.00 - 00.11.30
• As the virtual user went down from 50 to 5 at 00.07, the CPU usage doesn't go down but it stagnated. I expected the CPU usage would drop drastically similar to the number of virtual users

Things I Tried

• Disabling open telemetry exporter: I suspected the exporter may use some CPU too. However I didn't notice significant difference after disabling the open telemetry exporter.
• SSH into the container, install HTOP and directly see CPU usage. The CPU usage show on HTOP is almost meaningless here. It just goes up and down, I don't see any pattern.
• Run the app locally and run load test locally using similar setup but with different number of virtual user and duration. Result is as expected:
  

Question

• Is the CPU Usage pattern as expected?
• Is there anyway I can tweak the prometheus stack such that the gathered CPU usage will be similar to my expectation?

Additional Details

• Prometheus CPU Usage Query:

  sum(
    node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{cluster="$cluster", 
  namespace="$namespace"}
  * on(namespace,pod)
    group_left(workload, workload_type) 
  namespace_workload_pod:kube_pod_owner:relabel{cluster="$cluster", namespace="$namespace", 
  workload_type="$type"}
  ) by (workload, workload_type)
  

• Environment:
  • Kubernetes: GCP Kubernetes Engine v1.18.16-gke.502
  • Cluster: 2 node of n2d-high-cpu (4vCPU 4GB RAM)

[paulfantom]

It is important to note here that underlying recording rule responsible for node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate is using rate() over static 5m range. This function will take all samples from a given time range and return the average rate of change in a given time interval, hence all spikes will be smoothed out and you will get a different output than the one from htop. There is a function that would be similar (but not identical) to htop output and it is irate(). It will take only the last two samples from the selected time range, while discarding the rest of the samples, and calculate a per-second rate.

For this particular case, rate is better at detecting sustained CPU load and irate() would perform better for short-lived jobs.

There is a solution that would possibly allow us to marry both worlds. rate() will perform the same as irate() if given a small enough set of data and grafana allows using dynamic range intervals for their queries. However, we would need to stop using node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate recording rule and somehow move everything behind it to the dashboard query.

Note: all those recording rules and dashboards are coming from https://github.com/kubernetes-monitoring/kubernetes-mixin project, so it would be good to open an issue there.

More on the topic of rate() vs irate(): https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusRateVsIrate