container_spec_cpu_quota is missing

[jzeng4]

We use the cgroups metrics that is generated by cadvisor. Through our experiments, we found that there are some missing metrics for some of our application containers. Those metrics include:

• container_cpu_cfs_periods_total
• container_cpu_cfs_throttled_periods_total
• container_cpu_cfs_throttled_seconds_total
• container_spec_cpu_quota

It seems that all of them are related to cpu. I wonder if I need to specify some configurations to enable those metrics per application?

cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="Unknown",kernelVersion="5.4.189.1-rolling-lts-linkedin",osVersion="CentOS Linux 7 (Core)"} 1

[jzeng4]

We enabled the soft quota for cpu. Not sure if this matters?

[jzeng4]

I found the reasons why missing the above metrics:

If "cpu.cfs_quota_us" is "-1" (or say cpu soft quota is enabled), the cadvisor source code doesn't read the value. So "spec.Cpu.Quota" is zero in this case.

For the other missing metrics (container_cpu_cfs_periods_total, container_cpu_cfs_throttled_periods_total, container_cpu_cfs_throttled_seconds_total), since "spec.Cpu.Quota" is zero, those metrics are skipped, e.g. https://github.com/google/cadvisor/blob/master/metrics/prometheus.go#L193

[davidg-datascene]

We are hitting the same problem. For reference I've logged an issue with Azure as AKS is our cluster stack - Azure/AKS#3216 but the issue probably falls under this domain.

[lusson-luo]

I found the reasons why missing the above metrics:

If "cpu.cfs_quota_us" is "-1" (or say cpu soft quota is enabled), the cadvisor source code doesn't read the value. So "spec.Cpu.Quota" is zero in this case.

For the other missing metrics (container_cpu_cfs_periods_total, container_cpu_cfs_throttled_periods_total, container_cpu_cfs_throttled_seconds_total), since "spec.Cpu.Quota" is zero, those metrics are skipped, e.g. https://github.com/google/cadvisor/blob/master/metrics/prometheus.go#L193

so, how do you resolve?

[davidg-datascene]

Any update on this? Thanks

[lusson-luo]

@davidg-datascene i use kube_pod_container_resource_requests{resource="cpu"} replace container_spec_cpu_quota

[davidg-datascene]

@lusson-luo - Hi, thanks, but still need the throttled metrics i.e container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total.

[lusson-luo]

sorry, this two metrics i don't know

[lusson-luo]

@davidg-datascene i use kube_pod_container_resource_requests{resource="cpu"} replace container_spec_cpu_quota

i write error, use kube_pod_container_resource_limits{resource="cpu"} to replace container_spec_cpu_quota

[moritzschmitz-oviva]

For us the metric becomes empty (or not measured) when the container does not have a CPU limit. Does that make sense?

[davidg-datascene]

For us the metric becomes empty (or not measured) when the container does not have a CPU limit. Does that make sense?

I thought this at first but the deployment spec contains limits for CPU and tested changes to Mi values as well but made no difference.

spec of deployment:

resources:
    limits:
       cpu: "3"
       memory: 3Gi
    requests:
        cpu: "1"
        memory: 3Gi

The cluster is getting upgraded to AKS 1.24 in next few weeks and all the VM worker nodes will need replacing with a newer image/version. I'll check again when nodes are replaced to see if issue persists.

[davidg-datascene]

I've found where the issue lies in our case, at least. In our terraform that defines the Azure AKS node pools we set a kubelet_config.  For every node pool where this has been set we are missing the CFS quota metrics.  If I log into one of the nodes and check the kubelet config, I can see the value as false:

terraform yaml config for kubelet:

kubelet_config:
      container_log_max_size_mb: 100

cat ./etc/default/kubeletconfig.json | grep CFS
    "cpuCFSQuota": false,
    cat ./etc/default/kubeletconfig.json | grep -i log
    "containerLogMaxSize": "100M",   <-- we are only changing this value but cpuCFSQuota is false, hence no CFS stats

For where we are not setting a kubelet_config there is no  kubeletconfig.json file so accepts the defaults as per the Azure configuration - https://learn.microsoft.com/en-us/azure/aks/custom-node-configuration meaning CFS quota is true and you get the stats. The fix but not yet tested is we should be able to set https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster_node_pool [cpu_cfs_quota_enabled] in the terraform code to enable the CFS stats.

[ervinb]

Went deep on this, and I was banging my head why container_cpu_cfs_throttled_seconds_total is the only one missing while the others are there.

Turns out the Helm chart for kube-prometheus-stack drops that metric by default.

Just something to check for folks arriving from Google.

[tim-hilt]

@ervinb same for me! Lesson learned: Always check on the most fundamental level, which in this case was "are the limits actually set?"

Good idea to leave that information here.

If the cadvisor-metrics for limits are still not reliable enough, there are a few alternative ways that you can check to get the limits as a prometheus-metric:

• use kube-state-metrics
• The kubelet of your k8s cluster usually deploys a cadvisor-instance on its own
• The kube-scheduler exposes limits-metrics, however getting this to work is quite cumbersome, because the cli-arguments for the scheduler have to be adapted