Decreasing disk usage across the cluster

[kevlaube]

We have a cluster deployed with 15 brokers and would like to decrease the size of the disks on every broker across the cluster. We were hoping to do this by decreasing the global default storage across the cluster, setting the delete-claim setting to true, and then gradually replacing/adding brokers and rebalancing. This doesn't seem to work, after decreasing the default storage setting we are seeing the below message.

2022-10-24 20:48:27 WARN  AbstractModel:437 - Reconciliation #2(timer) Kafka(kafka/staging-cluster): Only the following changes to Kafka storage are allowed: changing the deleteClaim flag, adding volumes to Jbod storage or removing volumes from Jbod storage, changing overrides to nodes which do not exist yetand increasing size of persistent claim volumes (depending on the volume type and used storage class).
2022-10-24 20:48:27 WARN  AbstractModel:442 - Reconciliation #2(timer) Kafka(kafka/staging-cluster): The desired Kafka storage configuration in the custom resource kafka/staging-cluster contains changes which are not allowed. As a result, all storage changes will be ignored. Use DEBUG level logging for more information about the detected changes.

After attempting to add a new broker to see if it picked up the new storage setting, we have confirmed it does not. We understand that decreasing disk size is not supported by the underlying provider, but is it possible to change this default in a way where newly added brokers will pick up the new configuration for a smaller disk?

[scholzj]

Decreasing volume size is normally not supported by Kubernetes and the different infrastructures. So this will always require some manual workarounds which will always be risky and nasty.

If you use JBOD storage, you can add new smaller disk and then move the topics there using Kafka's reassignment script. And once you move them, you can remove the old disk. Otherwise, you will need to handle the deletions manually, but also take care that the new disks get synced in etc. before you delete the next one.

[kevlaube]

Got it, thanks. So there's no way to specify a smaller volume size for newer brokers if we are using the persistent volume storage?

[scholzj]

No, all the brokers use the same size storage.

[scholzj]

The current storage configuration is stored in an annotation on the Statefulset / StrimziPodSet. So you can manually change it there to get rid of the error you shared above. But this is basically hacking with the system. So you should first carefully test it somewhere before doing this in production ;-)