Status of Strimzi with Kraft?

[villesau]

Hi,

It looks like Kraft is production ready at least for new clusters: https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in

I'd like to understand what is the status with Kraft & Strimzi? Is it ready for prime time? Especially when setting things up from scratch.

[scholzj]

I guess the production readiness is a question of opinion as there are many missing features today which are quite important for many users such as SCRAM-SHA authentication, JBOD support, upgrades from ZooKeeper based clusters etc. Please search for the previous discussions here and on Slack for more details.

You can also check the strimzi/proposals#77 proposal and the video linked there for how it might look like when it is ready.

[scholzj]

Not sure I understand what are you trying to say.

[lucasfcnunes]

Oh.

I was addressing the following:

I'm in no official position to speak for the Apache Kafka project itself. There it is just my personal opinion.

by bringing up the official Apache Kafka project's position on KRaft:

• that it is GA/production ready since 2022-08
• that ~1 year they will start releasing without ZK (Kafka 4.0)

[scholzj]

Yeah, I know about all of that. that is why I made it clear that that is my opinion.

[bygui86]

@scholzj I see your opinion, but can you please share which features are missing in KRaft to be considered production ready?

[scholzj]

Most of them are repeated again and again. The only thing which changed really in 3.5.0 is SCRAM-SHa support. But migration is not production-ready, JBOD support is missing, the KRaft controllers are not really operationalized (no Admin API access / management => so it is really hard to operate them).

[vyom-soft]

Hello Scholzj,
We are on Strimzi 0.36 with default zookeeper. We would like to use Kraft. If I enable the UseKRaft feature
gate, would that be sufficient. ? To get rid of Zookeeper. Will the kafka-cluster work as usual ?
Can you please recommend how to switch to KRaft without having to uninstall the running strimzi-kafka-operater and without destroying the existing kafka cluster running on other NS which the operator monitors?

[scholzj]

@olucha I do not have any timeline for Strimzi. If you check our PRs, you will see that we are actively working on it. And if you look at the Kafka mailing lists, it seems clear that Kafka does not really have a timeline either. Keep in mind that Apache Kafka is still missing many important features as well.

I have around 6 months to evaluate the final architecture of the solution, and I could start with Strimzi if it's clear that this time window would be enough to have, at least, that upgrade support in place.

I would expect us to have the missing features in Strimzi earlier than in 6 months. But I'm not sure I would consider it production-ready by then neither in Strimzi nor in Kafka.

But let me ask you ... I do not know what solution are you building ... but what KRaft features do you require for it? What are the benefits KRaft gives you? Why do you so desperately need KRaft for it? Do you run any KRaft-based Kafka clusters in production to understand the limitations? In the end, it is always your decision and your risks to take of course. But insisting on deciding on this right now regardless the quality makes me curious.

[olucha]

Sorry @bygui86, I thought asking about the roadmap for the support of UseKraft feature gate was related with a discussion called "Status of Strimzi with Kraft". No problem to delete the question and redirect it to Slack.

[ppatierno]

@olucha I would keep this here tbh. It's related to the status, and now we are curious about the questions Jakub asked to you :-)

[olucha]

Hello @scholzj , and thanks for your answer.
I don't actively follow Kafka's development or mailing list, I just know the "official statement" about readiness of Kraft from both Apache and Confluent, although I have already read your previous answers about your concerns to the current degree of completion of Kraft support en Apache Kafka itself.
I have to recognize that it's hard to get the complete picture when there's a KIP-500 with an associated Jira ticket marked as complete, but a lot of other KIPs that span from that... that's my fault.
Returning to your questions, I'm not desperate to have Kraft in place :-), but my client sees it as an improvement over current setup with Zookeeper+Kafka, and wants to architect a solution based on it. This is a hard requirement not stated by me...
I have seen your PR #9364, but looking to https://github.com/strimzi/proposals/blob/main/061-kraft-upgrades-and-downgrades.md I'm not sure if there are too much things to do before that particular functionality, upgrades and downgrades, be complete.

[scholzj]

I have to recognize that it's hard to get the complete picture when there's a KIP-500 with an associated Jira ticket marked as complete, but a lot of other KIPs that span from that... that's my fault.

Yeah, unfortunately, there are many additional KIPs since KIP-500 dealing with various aspects of KRaft. So it is quite hard to track. Plus even the KIPs don't give you a complete picture. There are also bugs in the code. And there is also a lack of APIs for operationalizing (automating) the KRaft controllers or the migration process -> which is what slows us down because these things are complicated to work around.

Returning to your questions, I'm not desperate to have Kraft in place :-), but my client sees it as an improvement over current setup with Zookeeper+Kafka, and wants to architect a solution based on it. This is a hard requirement not stated by me...

I think it would still be good to understand the reasons. I see a lot of misunderstanding among users about what will it actually bring. Don't get me wrong - I think getting rid of ZooKeeper is not a bad thing and I will not shed a tear when it is gone. But the reality for most Strimzi users is that in production the ZooKeeper pods will be replaced by the KRaft controller pods and that is it. There might be some improvements to how much Kafka can scale. But most users do not run at the scale to really utilize them. So it would be still interesting to understand why they require KRaft.

I have seen your PR #9364, but looking to https://github.com/strimzi/proposals/blob/main/061-kraft-upgrades-and-downgrades.md I'm not sure if there are too much things to do before that particular functionality, upgrades and downgrades, be complete.

#9364 is the first part of the upgrades. There will be a second part. as well soon I currently expect that to land in the next Strimzi version. But there is also #9324 for the migration, #9146 (that is hopefully close to being merged as well - and is essential for the upgrades as well), etc. So there is more work in progress.