How to stop accepting messages instead of being dead when the disk is full?

[pkutakov]

Hi All,

If you will load a lot of data into Kafka and your storage space became 100% used, kafka pod just enters into "Error->CrashLoopBackoff->Running->Error" cycle. Is there any settings just to stop accepting messages when disk is near capacity instead of simply falling?

Best regards

[scholzj]

You should set the retention of your topics so that the disk does not get full. That is the only option offered by Kafka itself.

We have some work in progress on a quota plugin that might help with this in https://github.com/strimzi/kafka-quotas-plugin, but I do not think there is any exact timeline for when it will be available.

[pkutakov]

Thank you very much, I've tried quota plugin (it works pretty well) and it seems to be almost what I need. With one small exception - it stops producer to generate messages so finally producer throws timeout error. But from client point of view it will be better to generate KAFKA_STORAGE_ERROR to let producer know what is going on. But it looks like throttling mechanizm cannot do that or... maybe I didn't understand how it works...

[scholzj]

I do not think this is possible. The plugin cannot throw any kind of error. It just controls the Kafka's built-in quota capability to gradually set the quite to 0.