KafkaBridge - metrics

[kreuzj]

Hi there,
I'm sorry for bothering but I'm still unable to get Kafka Bridge metrics working. What I did:

1. set "enableMetrics: true" in KafkaBridge descriptor file (as per this example: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-bridge-metrics.yaml)
2. added metrics definition to Kafka metrics config map (as per this example: https://github.com/strimzi/strimzi-kafka-bridge/blob/main/src/main/resources/jmx_metrics_config.yaml)

We are running Kafka in AKS and scraping kafka metrics via ContainerInsights agent to LogAnalytics Workspace. When KafkaBridge starts I see following in the log:

2022-10-07 08:04:25 INFO  Application:66 - Strimzi Kafka Bridge 0.21.4 is starting
2022-10-07 08:04:25 INFO  Application:71 - Metrics enabled and exposed on the /metrics endpoint
2022-10-07 08:04:26 INFO  HttpBridge:183 - Starting HTTP-Kafka bridge verticle...

... but still cannot find the metrics. The issue looks very similar to this strimzi/strimzi-kafka-bridge#641 but even I read through it several times, I was unable to get KafkaBridge metrics working... Can you pls. help?

Thanks
Jan

[scholzj]

Sharing your full logs and full KafkaBridge resource would be a good start. You can also share what you get when you try to scrape the metrics your self with curl for example.

[kreuzj]

Hi Jakub, thanks for prompt reply, here is the requested info (I had to replace sensitive info with *****):

1. deployment descriptor:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "17"
  creationTimestamp: "2022-06-22T12:31:24Z"
  generation: 17
  labels:
    app.kubernetes.io/instance: kb-*******
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka-bridge
    app.kubernetes.io/part-of: strimzi-kb-*******
    strimzi.io/cluster: kb-*******
    strimzi.io/kind: KafkaBridge
    strimzi.io/name: kb-*******-bridge
  name: kb-*******-bridge
  namespace: kafka
  ownerReferences:
  - apiVersion: kafka.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: false
    kind: KafkaBridge
    name: kb-*******
    uid: *******
  resourceVersion: "30792846"
  uid: *******
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      strimzi.io/cluster: kb-*******
      strimzi.io/kind: KafkaBridge
      strimzi.io/name: kb-*******-bridge
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        strimzi.io/auth-hash: "2125510927"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kb-*******
        app.kubernetes.io/managed-by: strimzi-cluster-operator
        app.kubernetes.io/name: kafka-bridge
        app.kubernetes.io/part-of: strimzi-kb-*******
        strimzi.io/cluster: kb-*******
        strimzi.io/kind: KafkaBridge
        strimzi.io/name: kb-*******-bridge
    spec:
      containers:
      - command:
        - /opt/strimzi/bin/docker/kafka_bridge_run.sh
        env:
        - name: KAFKA_BRIDGE_METRICS_ENABLED
          value: "true"
        - name: STRIMZI_GC_LOG_ENABLED
          value: "false"
        - name: KAFKA_BRIDGE_BOOTSTRAP_SERVERS
          value: *******:9096
        - name: KAFKA_BRIDGE_ADMIN_CLIENT_CONFIG
        - name: KAFKA_BRIDGE_CONSUMER_CONFIG
        - name: KAFKA_BRIDGE_PRODUCER_CONFIG
        - name: KAFKA_BRIDGE_ID
          value: kb-*******
        - name: KAFKA_BRIDGE_HTTP_ENABLED
          value: "true"
        - name: KAFKA_BRIDGE_HTTP_HOST
          value: 0.0.0.0
        - name: KAFKA_BRIDGE_HTTP_PORT
          value: "8080"
        - name: KAFKA_BRIDGE_CORS_ENABLED
          value: "false"
        - name: KAFKA_BRIDGE_AMQP_ENABLED
          value: "false"
        - name: KAFKA_BRIDGE_TLS
          value: "true"
        - name: KAFKA_BRIDGE_TRUSTED_CERTS
          value: *******/ca.crt
        - name: KAFKA_BRIDGE_TLS_AUTH_KEY
          value: *******/user.key
        - name: KAFKA_BRIDGE_TLS_AUTH_CERT
          value: *******/user.crt
        image: quay.io/strimzi/kafka-bridge:0.21.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthy
            port: rest-api
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kb-*******-bridge
        ports:
        - containerPort: 8080
          name: rest-api
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: rest-api
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /opt/strimzi/custom-config/
          name: kafka-metrics-and-logging
        - mountPath: /opt/strimzi/bridge-certs/*******cluster-ca-cert
          name: *******cluster-ca-cert
        - mountPath: /opt/strimzi/bridge-certs/*******-01
          name: *******-01
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kb-*******-bridge
      serviceAccountName: kb-*******-bridge
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 1Mi
        name: strimzi-tmp
      - configMap:
          defaultMode: 420
          name: kb-*******-bridge-config
        name: kafka-metrics-and-logging
      - name: ext-kafka-cluster-cluster-ca-cert
        secret:
          defaultMode: 292
          secretName: *******-ca-cert
      - name: *******-01
        secret:
          defaultMode: 292
          secretName: *******-01
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-07-05T10:33:01Z"
    lastUpdateTime: "2022-07-05T10:33:01Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-06-22T12:31:24Z"
    lastUpdateTime: "2022-10-07T14:28:12Z"
    message: ReplicaSet "kb-*******-bridge-bd8c9fc5f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 17
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

2. full log (containing KB startup):

#TLS/SSL
kafka.ssl.truststore.location=/tmp/strimzi/bridge.truststore.p12
kafka.ssl.truststore.password=
kafka.ssl.truststore.type=PKCS12
kafka.ssl.keystore.location=/tmp/strimzi/bridge.keystore.p12
kafka.ssl.keystore.password=
kafka.ssl.keystore.type=PKCS12

#Apache Kafka AdminClient

#Apache Kafka Producer

#Apache Kafka Consumer

#HTTP configuration
http.enabled=true
http.host=0.0.0.0
http.port=8080
http.cors.enabled=false
http.cors.allowedOrigins=
http.cors.allowedMethods=

++ dirname /opt/strimzi/bin/docker/../kafka_bridge_run.sh
+ MYPATH=/opt/strimzi/bin/docker/..
+ '[' -z -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties ']'
+ JAVA_OPTS='  --illegal-access=deny -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom'
+ exec java --illegal-access=deny -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -classpath '/opt/strimzi/bin/docker/../../libs/*' io.strimzi.kafka.bridge.Application --config-file=/tmp/kafka-bridge.properties
2022-10-07 14:26:40 INFO  Application:66 - Strimzi Kafka Bridge 0.21.4 is starting
2022-10-07 14:26:40 INFO  Application:71 - Metrics enabled and exposed on the /metrics endpoint
2022-10-07 14:26:41 INFO  HttpBridge:183 - Starting HTTP-Kafka bridge verticle...
2022-10-07 14:26:41 INFO  AdminClientConfig:372 - AdminClientConfig values: 
    bootstrap.servers = [***********:9096]
    client.dns.lookup = use_all_dns_ips
    client.id = 
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = /tmp/strimzi/bridge.keystore.p12
    ssl.keystore.password = 
    ssl.keystore.type = PKCS12
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = /tmp/strimzi/bridge.truststore.p12
    ssl.truststore.password = 
    ssl.truststore.type = PKCS12

2022-10-07 14:26:41 INFO  AppInfoParser:119 - Kafka version: 2.8.1
2022-10-07 14:26:41 INFO  AppInfoParser:120 - Kafka commitId: 839b886f9b732b15
2022-10-07 14:26:41 INFO  AppInfoParser:121 - Kafka startTimeMs: 1665152801748
2022-10-07 14:26:41 INFO  HttpBridge:104 - HTTP-Kafka Bridge started and listening on port 8080
2022-10-07 14:26:41 INFO  HttpBridge:105 - HTTP-Kafka Bridge bootstrap servers **************
2022-10-07 14:26:41 INFO  Application:224 - HTTP verticle instance deployed [c8a835a2-af84-4397-8e9e-1d7f96e79e50]

3. I tried to get metrics when connected to running KafkaBridge pod (ssh) and run

curl localhost:9404/metrics

... the response was:

curl: (7) Failed to connect to localhost port 9404: Connection refused

Could it be that the problem is because the config map used in the deployment descriptor doesn't exist (at least I cannot see it in my AKS namespace):

  - configMap:
          defaultMode: 420
          name: kb-*******-bridge-config
        name: kafka-metrics-and-logging

???

[scholzj]

Thanks. Can you share the actual KafkaBridge custom resource? That is what would allow to reproduce it.

That said, I do not think the Bridge metrics are on port 9404. As the bridge already has a HTTP server on port 8080, it is using that. So try curl localhost:8080/metrics.

[scholzj]

Could it be that the problem is because the config map used in the deployment descriptor doesn't exist (at least I cannot see it in my AKS namespace)

If it would not exist, then the pod would not start but it will be waiting for it. So I assume it is somewhere there.

[kreuzj]

do you mean KafkaBridge CRD? If so it is here: KB-CRD.txt...

However when tried curl localhost:8080/metrics it returned metrics... So the problem seems to be resolved - I just have to add the URL for scraping to LogAnalytics... Thanks for prompt reaction and you help!

[scholzj]

Great, glad we solved it.