User creation and syncing from Kafka

[rajjaiswalsaumya]

We have a requirement where our Kafka is externally exposed and the onboarding of Users happens when a customer from UI chooses to use the event stream. We want to create a user for this customer in Kafka on the fly. We used Kafka's admin client but the SCRAM-SHA login does not seem to work. After reading some docs, it seems Strimzi does not manage the syncing of users from Kafka to CRD while the same works for topics. Due to this, the user secret does not get created and and the login does not work. Is there a way we can make this work? We are using strimzi 0.32.1 chart.

[scholzj]

If you want to manage your Kafka users your self, you can disable the User Operator and use the Kafka Admin API only.

[rajjaiswalsaumya]

We don't want to disable the User Operator as we don't want to lose the benefit provided by the USER operator such as TLS secrets creations. As SCRAM is for only external customers, internal service users still require all those TLS creations. Due to this, with the user operator when we tested, we found Strimzi deletes users created in Kafka using Admin Client as CRDs are not present for those users.

On a side note, we started using Strimzi java Client and creating CRD which actually caters to the scenario of user creation. Due to this, a secret also gets created and the user is able to log in. Can you suggest if there are any other better ways?

[scholzj]

Yes, for the User Operator the custom resources are the single source of truth. So it will delete any users you create manually. There is no way to work around it. I guess the only way is to change your UI to create the KafkaUser resources in Kubernetes instead of using the Kafka Admin API.

[rajjaiswalsaumya]

Thanks @scholzj This helps. So the way we changed yesterday is using strimzi java api, and creating Cr. UI calls the API which in turn will trigger this client and create a CR. Strimzi now sinks this to Kafka. One query, If the password of the user is updated and if we update in Strimzi's CR for Kafka User and update secret, will Strimzi still go and sync this user to Kafka? I mean does it check if the password is updated ? If yes, how does it know that the User in Kafka has different credentials than what is present in CR?

[scholzj]

Sorry, I'm not sure I follow. When you create the KafkaUser resource for SCRAM-SHA-512 user, it normally generates the password. If you want to set your own password, you can configure it this way: https://strimzi.io/docs/operators/latest/full/deploying.html#custom_password_configuration