[Bug]: Improve performance of cluster

[ducle-ttlab]

Bug Description

I have create cluster kafka with strimzi with change many way from increase resources or replicas but when i'm testing performance with kafka-producer-perf-test.sh number it's doesn't change alway around 359324.469996 records/sec how can i improve this to procedure 1 million records ?

My command testing

/opt/kafka/bin/kafka-producer-perf-test.sh \
    --topic my-topic \
    --num-records 1000000 \
    --record-size 300 \
    --throughput -1 \
    --producer.config /tmp/perf.properties \
    --producer-props \
        batch.size=200000 \
        linger.ms=100 \
        compression.type=lz4 \
        acks=1

My config

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.6.0
    replicas: 3
    resources:
      requests:
        memory: 16Gi
        cpu: "8"
    listeners:
      - name: plain
        port: 9093
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 1Gi
        deleteClaim: false 
  zookeeper:
    replicas: 5
    resources:
      requests:
        memory: 16Gi
        cpu: "8"
    jvmOptions:
      -Xms: 4096m
      -Xmx: 4096m
    storage:
      type: persistent-claim
      size: 1Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Steps to reproduce

No response

Expected behavior

Improve procedure 1 million records/second

Strimzi version

latest

Kubernetes version

1.28.1

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

The performance of your cluster depends on many factors. From the resources you assigned to it, to the networking of your cluster, storage performance etc. Each infrastructure is a little bit different, so it is hard to comment on any specifics - you need to look at each of the aspects and see how it does (for example in your case, you seem to have lot of CPU compared to the amount of memory from what I usually see).

A lot also depends on the use-case you are trying to do:

• How did you configure the topics you use (partitions, replicas, min-in-sync-replicas etc.)
• How do you run the client -> do you really need 1M 300-byte messages per second from a single producer as you seem to suggest? Is acks=1 really the reliability you want?

For a lot of these things, it is also important to understand how Kafka works (e.g. to know when increasing number of brokers does or does not help etc.). So make sure you have some understanding of that as well.

[ducle-ttlab]

I'm using benchmark on AWS using gp3 for storageclass with throughput 125MiB/s.
My topic is create is

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 100
  replicas: 3
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824

do you really need 1M 300-byte messages per second from a single producer as you seem to suggest? -> I'm just make sure kafka can process with 1M 300-byte messages per second from multiple producer.
acks=1 => I'm just follow guide to increase performance

[ducle-ttlab]

Can you give me some usecase config can process with 1M 300-byte messages per second from multiple producer.

[scholzj]

I'm just make sure kafka can process with 1M 300-byte messages per second from multiple producer.

But your question suggests you have only one producer in your test. So, have you tried to run more of them in parallel?

acks=1 => I'm just follow guide to increase performance

And do you really understand what acks=1 means for reliability? Is that acceptable for your application? Is it fine for it to lose messages in some scenarios?

Can you give me some usecase config can process with 1M 300-byte messages per second from multiple producer.

I cannot do that, sorry. As I said, you have to consider your environment and your own use case. I do not have such environment nor such use case to just give you some configuration.