Very bad performance during consumer perf test

[rgarcia89]

Describe the bug
We are currently reviewing the usage of this operator since we are currently running kafka locally installed on virtual machines.
So far the installation of the operator as well as the cluster has been very smooth. As next step we wanted to run some performance tests. When trying to do so we tried to run some consumer performance tests with kafka-consumer-perf-test.sh. Unfortunantely they all end up in a timeout. I am wondering what could be the reason for this bad performance and hope to find here some help.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Operator with helm chart.
2. Deploy kafka cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: sisr-cluster
spec:
  kafka:
    version: 3.2.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    jvmOptions:
      "-Xmx": "2g"
      "-Xms": "2g"
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 80Gi
        class: default
        deleteClaim: true
  zookeeper:
    replicas: 3
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    jvmOptions:
      -Xms: 512m
      -Xmx: 512m
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: true
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5

3. Create a topic
   kubectl run kafka-producer -ti --image=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 --rm=true --restart=Never -- bin/kafka-topics.sh --bootstrap-server sisr-cluster-kafka-bootstrap:9092 --create --replication-factor 3 --partitions 100 --topic rgarcia
4. Run a consumer perf test against the topic
   kubectl run kafka-consumer-perf -ti --image=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 --rm=true --restart=Never -- bin/kafka-consumer-perf-test.sh --bootstrap-server sisr-cluster-kafka-bootstrap:9092 --topic rgarcia --messages 100
5. See error
   You will see the request running into a timeout because the cluster did not complete it in time

kubectl run kafka-consumer-perf -ti --image=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 --rm=true --restart=Never -- bin/kafka-consumer-perf-test.sh --bootstrap-server sisr-cluster-kafka-bootstrap:9092 --topic rgarcia --messages 100
If you don't see a command prompt, try pressing enter.
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
WARNING: Exiting before consuming the expected number of messages: timeout (10000 ms) exceeded. You can use the --timeout option to increase the timeout.
2022-07-20 14:37:31:309, 2022-07-20 14:37:41:426, 0.0000, 0.0000, 0, 0.0000, 3448, 6669, 0.0000, 0.0000
pod "kafka-consumer-perf" deleted

Expected behavior

Environment (please complete the following information):

• Strimzi version: 0.29.0
• Installation method: Helm chart
• Kubernetes cluster: 1.23.8
• Infrastructure: AKS

[scholzj]

How are you producing the messages? IIRC the consumer will just consume the messages, but you are not producing them anywhere (maybe you just forgot to add the step here, not sure). Normally the kafka-producer-perf-test.sh would do it.

[rgarcia89]

You are totally right @scholzj ...
I was able to run the perf test successfully now, Unfortunantely the performance is far away from what we are getting on the VMs on which kafka is deployed currently as docker containers.

Running against on VMs deployed kafka

./kafka-consumer-perf-test.sh --topic raulgs --bootstrap-server=$SERVER --messages 100000 --timeout 1000000
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-07-21 09:25:54:970, 2022-07-21 09:25:56:057, 97.6563, 89.8402, 100000, 91996.3201, 371, 716, 136.3914, 139664.8045

Running against Kafka running on kubernetes

 kubectl run kafka-consumer-perf -ti --image=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 --rm=true --restart=Never -- bin/kafka-consumer-perf-test.sh --topic internal --bootstrap-server=sisr-cluster-kafka-bootstrap:9092 --messages 100000 --timeout 1000000
If you don't see a command prompt, try pressing enter.
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-07-21 09:26:40:121, 2022-07-21 09:26:44:708, 97.6816, 21.2953, 100026, 21806.4094, 3848, 739, 132.1808, 135353.1800
pod "kafka-consumer-perf" deleted

All of them are using the same disk type in azure.

[scholzj]

TBH, performance is often the question of the infrastructure. Without knowing your infrastructure, it is hard to comment on it. There are many things which can affect it:

• Performance of the infrastructure (networking, storage IO, etc.)
• SDN used on your Kuber cluster
• Amount of available resources (you did not configure resources in your Kafka CR you shared earlier)

It is also not clear in which oder you run the producers and consumers (in parallel? serially?) and whether the data were cached etc. So it is hard to comment on the numbers.

From my experience, 100k messages (~100MB of data) is hardly something what shows a performance. With the speeds you got you ended in few seconds. If you compare the numbers, you will see that the fetch speeds were similar, but the rebalance took almost 4 out of the 5 seconds it took on Kubernetes. In any case, using more messages to make the test run longer will give you more meaningful numbers.

[rgarcia89]

You right using more messages changed the speed dramatically.

Running against Kafka on vm

./kafka-consumer-perf-test.sh --topic $TOPIC --bootstrap-server=$SERVER --messages $RECORDS --timeout 1000000
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-07-21 11:58:24:841, 2022-07-21 11:58:27:720, 976.3714, 339.1356, 1000363, 347468.9128, 373, 2506, 389.6135, 399187.1508

Running against Kafka on kubernetes

I have no name!@kafka-rg:/bin$ kafka-consumer-perf-test.sh --topic $TOPIC --bootstrap-server=$SERVER --messages $RECORDS --timeout 1000000
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2022-07-21 13:00:01:201, 2022-07-21 13:00:07:345, 953.6743, 155.2204, 1000000, 162760.4167, 3663, 2481, 384.3911, 403063.2809

Of course there are a lot variables that could be having impact here. I am just wondering are there any kind of tuning parameters you would recommend for a cluster like the one I have build?