Kafka Strimzi could handle maximum of partition

[chary1112004]

Hi,

We are doing test relate to maximum of partition of kafka strimzi.

We have used kafka in aws. Each instance type of msk, there is very limit for partitions. If the partitions reach near the limit then it makes kafka is down and it leads services could not connect to kafka. Refer from https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html

As we check apache kafka itself could handle up to 200k partitions https://www.confluent.io/blog/apache-kafka-supports-200k-partitions-per-cluster/

Does the kafka strimzi could handle maximum number of partitions as apache kafka (200.000 partitions)?

Any information would be very much appreciated.

Thanks!

[scholzj]

Strimzi does not have any hard limits like that and Apache Kafka itself has no hard limits either (and I would not rely too much on a single number from a blog post from 2018 - although my guess is that the development over the 5 years should tend to increase this number rather than decrease).

The actual limits will depend heavily on your environment -> hardware, networking, things such as how good noisy neighbor isolation it has etc. So you would basically need to measure it yourself in your own environment. Strimzi itself does not modify Kafka in any way or interfere with how Kafka works. So I would not expect the results with Strimzi to be any different from any other Apache Kafka installation in the same environment.

[chary1112004]

Hi @scholzj,

Thank you for the information. Now we are testing up to 2.000 partitions and services still could connect to the strimzi kafka.

It increases quite amount of cpu and memory for kafka, it is stable cpu and memory for zookeeper (topics are just created, not in use)
Kafka
Before stress test | After stress test
CPU 24m | CPU 778m
Memory 2701M | Memory 3375M

Zookeeper
Before stress test | After stress test
CPU 10m | CPU 28m
Memory 664M | Memory 672M

However it seems there are too many metrics when partitions/topics increase then it makes prometheus could not pull metric from kafka. Action needed is increasing scrapeTimeout then it is ok.

Thanks!

[scholzj]

Yeah, the JMX Prometheus Exporter starts to struggle a bit with to many metrics. You might want to use the configuration of the metrics to filter them out a bit more.

[chary1112004]

Yes, thanks for the suggestion!