Upgrade Existing Cluster to use Mutual TLS/Authorization

[mohitruhaan]

Hi Experts,

We want to upgrade Production Clusters(Strimzi 0.27.0--Kafka 2.8.0) to enable Mutual TLS and Authorization between Kafka Brokers/Clients and are working on Migration strategy to have minimum/no impact for the Customers. Currently the Partners are connecting to 9094 Listener using Server Authentication.

Below are the Upgrade steps:

1. Add a new Node-Port External Listener (9095) in existing cluster and configure it with authentication type TLS.

2. Create KafkaUsers for Partners and share certificate/password details with them. Ask them to test connection with Listener Port 9095 and Produce/Consume some test messages.

Now the Server+Client Authentication should be working fine. But how should we enable Authorization with no impact?

1. Can we enable Authorization only on a specific Listener 9095? Or will it be applied to all the Listeners?
2. Kafka has parameter : "allow.everyone.if.no.acl.found”, how can we use it in strimzi to avoid Authorization errors?

Thanks,
Mohit

[scholzj]

Not sure there is an easy answer to all your questions. I think going through it with a new listener is the best way possible, you can move them gradually. When you talk about partners, it sounds like external entities. One thing to be aware of is that the mTLS authentication does nto handle any certificate revocations currently or anything like that. So it is not completely easy to get rid of the partners while keeping others. The only 100% clean way is really to replace the whole CA and update everyones certificates. So I think you should also consider something like SCRAM-SHA-512 authentication where ou can simply just delete the user.

As for the authorization, I think you have two options:

• If you use no authentication and authoirzation today, maybe you can first migrate all users to the new listener with authentication and only then enbale the Authorization for all of them.
• You could also use the authorization for the new users on new listener and set up special rules for the ANONYMOUS user how the clients on the old listener will be treated. You can eiter selectively allow them to do some operations like reading from some topic or you can make the ANONYMOUS user a super user whio can do everything (sounds insecure, but if I got it right this is exactly how they are treated today).

[mohitruhaan]

Thanks @scholzj !

By Partners/Clients i mean different teams who are using the Kafka Service owned by us.
I don't find Challenge in Authentication as I had done POC with Cert-Manager which will act as a CA for Broker and Clients. So on new Listener Port 9095, I will get a Certificate Signed by Cert-Manager and tag it on Listener and provide its Certs to all Clients for Server Authentication.
Only Challenge i see is Authorization for which the best possible Solution i can think of is:

• Create a Super-User and enable only Authentication on 9095 in Kafka. Ask all the Clients to create their keystore with Super-User creds.
• Meanwhile existing traffic keeps running on old listener-9094 with Super-User in their keystores.
• Once Authentication is fine, we will upgrade Kafka with Authorization which should not be of concern to Clients as they are using Super-User privileges.
• Meanwhile, we will create a Kafka-User per Client with the ACLs they need, and ask them to test using their Kafka-User and if fine, we will request them to gradually switch from Super-User to their respective Kafka-Users.
• Delete Super-User once all Clients have migrated to Kafka-User.

What do you think about this Strategy?

Thanks again for your valuable inputs. :)

[scholzj]

I think that sounds reasonable.