How do I set CPU utilization quotas?

[jrivers96]

The definition for CPU utilization quotas is listed below?

CPU utilization quotas specify a window for broker requests from clients. The window is the percentage of time for clients to make requests. A client makes requests on the I/O threads and network threads of the broker.

Given a specific user ( or even a single client ), how do I know how much CPU is being utilized for that client? I can't seem to find a metric that tells me this.

[scholzj]

TBH, I'm not aware of any metric for this. That does not mean it does not exist, but it is definitely not on any of our dashboards.

[jrivers96]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas

@scholzj Any idea how I might add the below request-time metric?

Two new metrics and corresponding sensors will be added to track request-time and throttle-time of each quota entity for quota type Request.  The request-time sensor will be configured with the quota for the user/client so that quota violations can be used to add delays to the response.

https://github.com/apache/kafka/blob/99b9b3e84f4e98c3f07714e1de6a139a004cbc5b/core/src/main/scala/kafka/server/ClientRequestQuotaManager.scala#L82

[scholzj]

I guess you would need to edit the JMX Exporter configuration (the YAML file in the config map in the recent versions) to match the JMX metric and convert it to Prometheus metric. That configures which metrics will be exposed.

[jrivers96]

FYI - A metric that is critical for the new quota setting that strimzi added for KafkaUser is the below. I'm currently trying to figure out how to turn it on. May want to make this default?

Request quota metrics per (user, client-id), user or client-id

kafka.server:type=Request,user=([-.\w]+),client-id=([-.\w]+)

Two attributes. throttle-time indicates the amount of time in ms the client was throttled. Ideally = 0. request-time indicates the percentage of time spent in broker network and I/O threads to process requests from client group. For (user, client-id) quotas, both user and client-id are specified. If per-client-id quota is applied to the client, user is not specified. If per-user quota is applied, client-id is not specified.

[scholzj]

I don't have a strong opinion on how important this is TBH. I guess it should not be a problem to add it to the examples if you think it should be there.

[jrivers96]

How would you set the request percentage quota without having that metric? You'd have to estimate it by gradually setting the request percent lower until you started getting throttled?

[scholzj]

I don't have an answer since I don't think I ever used it apart from testing that the value is passed to Kafka TBH ;-)

[jrivers96]

If I curl 127.0.0.1:9404, inside the kafka broker pod, do I get all the kafka broker metrics or only the ones that we are turned on in the strimzi metric config map?

[scholzj]

No, these are only the exposed ones => i.e. those matching the rules in the config map.