[DPE-7148] feat: KIP-714 PoC

[imanenami]

Description

This PoC PR implements KIP-714 central client metric aggregation feature by adding a CustomMetricsReporter class. Ideally, this class should be bundled with the snap. This class is a simple implementation of org.apache.kafka.server.telemetry.ClientTelemetryReceiver, which receives metrics from clients, and then forwards them to the OTelCol HTTP port, which would then push them to COS prometheus.

To test this PR:

Build & Deploy

charmcraft pack
juju add-model kip714
juju deploy ./*.charm -n 3 --config roles="broker,controller"
juju deploy opentelemetry-collector --channel 2/stable otelcol

Consume COS offers and Integrate

Assuming you have COS deployed:

K8S_CONTROLLER="k8s"
juju consume $K8S_CONTROLLER:admin/cos.grafana-dashboards
juju consume $K8S_CONTROLLER:admin/cos.loki-logging
juju consume $K8S_CONTROLLER:admin/cos.prometheus-receive-remote-write

juju integrate otelcol kafka
juju integrate otelcol loki-logging
juju integrate otelcol grafana-dashboards
juju integrate otelcol prometheus-receive-remote-write

Configure client metric collection configuration

I used kafka-configs.sh because kafka-client-metrics.sh is not bundled in our snap. I think we should do that:

# Producer metrics
juju ssh kafka/0 'sudo charmed-kafka.configs \
    --bootstrap-server $(hostname -i):19093 \
    --command-config $CONF/client.properties \
    --alter \
    --entity-type client-metrics \
    --entity-name producer-metrics \
    --add-config "metrics=org.apache.kafka.producer.,interval.ms=1000"'

# Client metrics
juju ssh kafka/0 'sudo charmed-kafka.configs \
    --bootstrap-server $(hostname -i):19093 \
    --command-config $CONF/client.properties \
    --alter \
    --entity-type client-metrics \
    --entity-name consumer-metrics \
    --add-config "metrics=org.apache.kafka.consumer.,interval.ms=1000"'

Run a perf test (or any other CLI-based client activity)

juju ssh kafka/0 'sudo charmed-kafka.topics \
    --bootstrap-server $(hostname -i):19093 \
    --command-config $CONF/client.properties \
    --create --topic test'

juju ssh kafka/0 'sudo charmed-kafka.producer-perf-test \
    --producer.config $CONF/client.properties \
    --producer-props bootstrap.servers=$(hostname -i):19093 \
    --throughput 1000 \
    --num-records 5000 \
    --record-size 1024 \
    --topic test \
    --print-metrics'

Check prometheus

The metrics are available under org_apache_kafka_producer_* and org_apache_kafka_consumer_* prefixes:

Checklist

• I have added or updated any relevant documentation.
• I have cleaned any remaining cloud resources from my accounts.

[marcoppenheimer]

So lots of thoughts here.

1. I can't see the source of the bundled JAR. I agree that having the add-on in the snap is the way to go though
2. Yes, definitely adding the kafka-client-metrics.sh to the snap apps is good
3. It's possibly premature, but I believe that ClientTelemetryReceiver is to be deprecated in 4.2 - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1217:+Include+push+interval+in+ClientTelemetryReceiver+context. Do we think it's worth waiting for this before continuing?
4. It's probably defaults somewhere, but I'm not quite sure how your instructions work for collecting data from arbitrary clients. Wouldn't you need a match config?
   • Additionally, what's the UX here for safely matching clients do you think? Charm config?