How to configure TLS on topic consumer side only

[kingnarmer]

I have k8s logging setup where logs --> fluentd --> kafka then logstash pulls logs from kafka. I would like to keep sending logs fluentd --> kafka without authentication but require authentication on logstash side pulling logs from kafka.

Is there instructions on how to accomplish this ?

Thanks

[scholzj]

You can configure different listeners with different authentication options. You can check the docs for the details. But be careful about authorization if you plan to use it as Kafka supports it only cluster-wide and not per listener.

[kingnarmer]

@scholzj I looked at documents and couldn't find how to allow UN-authenticated clients (producers) to send logs while restricting consumer to use tls.

End goal is to allow producers to send logs say to port 9094 without authentication and restrict consumers to use cert to pull logs say on port 9095 .

How to tell listener type of traffic allowed ?

Thanks

[scholzj]

First of all, it is probably a bad idea 😉. What if someone starts sending you TBs of data? It will be constantly filling up your disk, you will possibly need to pay for the data transfers, etc.

If you really want to do it ... It is not easy and straightforward. It is more of a hack than anything else.

1. Setup two listeners:
   • one with authentication enabled
   • one with authentication disabled
2. Enable type: simple authorization.
3. Use the Kafka Admin API to give a user ANONYMOUS ACL rights to publish to the topic which you want to use. This has to be done using the Kafka Admin API or CLIs. You cannot use the User Operator for it.
4. Create users for your consumers with regular ACLs for consuming. This can be done using the User Operator.
5. Have the producer use the listener without authentication and produce the messages
6. Have the consumers use the listener with authentication to consume the messages

[kingnarmer]

Thanks. Will test it and see how it goes. If I enable TLS on producer side , which authentication method is the most stable to use with log producer like fluetd.

[scholzj]

Sorry, stable in what sense? SCRAM-SHA-512 passwords have no expirations and you can set the password in any way you want. TLS certificates have expirations, renewals, etc. Not sure if this is the stability you meant or not.