Switching storage class via node deletion

[mdharper]

I haven't tried at this point, but I was wondering:
Is it possible to switch Kafka cluster storage on a per-node basis using the strimzi.io/delete-pod-and-pvc=true annotation?

I'm thinking something like this may work:

1. Change per-node storage class via override in the Kafka resource definition
2. Cluster operator logs complain storage options can't be changed once pvc is created
3. Apply annotation to a node using the "old" storage class to delete the pod/pvc
4. Pod and pvc recreated using specified storage class

I'd appreciate any feedback on whether this is currently possible. If not, it seems like it'd be a useful feature.

[scholzj]

Almost ... I think this is missing one step. The storage configuration is stored in an annotation of the StatefulSet (Kafka or ZooKeeper, depending on which one you are changing). It is compared between the annotation and the Kafka CR and if unsupported changes are made, they are ignored and the correct configuration from the StatefulSet is used. So I think you would need to do this:

1. Change per-node storage class via override in the Kafka resource definition
2. Change the annotation to match the Kafka custom resource
3. Apply annotation to a node using the "old" storage class to delete the pod/pvc
4. Pod and pvc recreated using specified storage class

I never tried it, but I think this should work.