Kafka broker (trace) logs

[kreuzj]

Hello, is there any way how to get kafka broker trace logs? I mean logs which I get when I execute:

kubectl logs <kafka-brk-name> -n kafka

We are using K9s for managing/troubleshooting kafka cluster but the logs usually rotates quite fast so we are unable to dive into the history deep enough. Alternatively is there any possibility to redirect such logs to any external tool for further troubleshooting? Can you advise what is the recommended way to work with logs (for troubleshooting purposes)?

Thanks&Regards
Jan

[scholzj]

Normally, your Kubernetes cluster should have some logging stack that will be collecting and store the logs long term. The container engine normally captures the logs from the STDOUT and STDERR and stores them in files. These files are scanned by tools such as Fluentd, Fluentbit, Logstash, and others and transferred into some kind of database where you can access them (such as OpenSearch / ElasticSearch etc.).

The kubectl logs command basically gets you the files from the container engine which are essentially just a buffer - so they have typically a limited size and limited time for which they are stored.

[kreuzj]

Hello Jakub, thanks for explanation. We are using AKS and we know there is a possibility to forward logs to Azure log analytics workspace. But in fact we didn't setup this way of log forward for kafka as LA is quite expensive if there is big ingest (which is expected in case of kafka logs)...
Would it be however possible to access these logs somewhere on the filesystem? I mean to connect to AKS node and find the log on the filesystem - would such approach make sense/be possible?

Thks Jan

[scholzj]

Would it be however possible to access these logs somewhere on the filesystem? I mean to connect to AKS node and find the log on the filesystem - would such approach make sense/be possible?

They would be somewhere on disk. But the exact path would depend on your setup. Also, it will IMHO not give you anything more than kubectl logs.