old data in the PVC has been restored

[hari819]

@scholzj

we are running 3 brokers with 3 zookeeper , using the 0.27 version of the operator ,

this is the volume section of the Kafka broker in the cluster yaml ,

    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          class: SSD
          size: 500Gi
          deleteClaim: false
        - id: 1
          type: persistent-claim
          class: SSD
          size: 500Gi
          deleteClaim: false

last week when the data was almost full , 400GB , i stopped the cluster operator first , scaled down ZK , Kafka .

Later I deleted all the PVCs for Kafka and Zookeeper to start the cluster as a fresh one , it was all good and data started accumulating when we started using the fresh kafka cluster, but today I noticed that all the old data which got deleted when i deleted the PVC got restored again , and the PVC is almost full again ,

is this expected with my setting of deleteClaim as "false" , please could you help me on this ?

[scholzj]

The deleteClaim defines whether the PVCs should be deleted when the Kafka cluster is deleted. From your description, you did not do that -> you just stoped the operator and scaled down the clusters. So this has probably nothing to do with this.

When you delete the PVC as you described, it is up to your Storage Class and volume configuration what happens with the PVs. Whether they are deleted as well or whether they are kept. After you delete the PVCs and restart the Strimzi operator, it will create new PVCs. And your storage class / storge provisioner will bind it to some PVs. Normally, these should be fresh empty PVs so you should start with an empty slate.

TBH, I do not know what happened in your case. My best guess is that either you didn't deleted the PVCs (ir might take time etc.). Or maybe you did but your storage provisioner bound the new PVCs to the old PVs? Strimzi itself does not directly interact with the PVs and with the data, so I think it must have been something like that. Keep in mind that with Kafka replication, even one broker which keeps the data might replicate it to the others again over time.

[hari819]

I did see the PVCs start fresh from like 15GB on the recreation of the cluster 5 days back, today also for some time the data was only 72GB , but by EOD our time , i saw the data is 350GB and i see old topics/data which i deleted last time ,

this is very confusing , how can this happen ?

[scholzj]

Well, as I said, most likely is that one of them was not deleted properly and the rest got resynced from it. I don't think I can help much more. I do not know what kind of storage are you using, how is it provisioned nothing. So it is hard to say. As I said before, Strimzi just creates the PVCs. The PVs bound to it are infrastructure thing.

[hari819]

Thankyou @scholzj , I will verify on the storage part ,

[hari819]

@scholzj the operator logs show some patching on PVC , during the same time , will they be useful in any way?

2022-04-12 23:21:09 DEBUG AbstractResourceOperator:115 - Reconciliation #4089(watch) Kafka(kafka/kafka): ### PersistentVolumeClaim kafka/data-0-kafka-kafka-0 already exists, patching it 2022-04-12 23:21:09 DEBUG ResourceDiff:33 - Reconciliation #4089(watch) Kafka(kafka/kafka): Ignoring PersistentVolumeClaim data-0-kafka-kafka-0 diff {"op":"remove","path":"/metadata/annotations/pv.kubernetes.io~1bind-completed"} 2022-04-12 23:21:09 DEBUG ResourceDiff:33 - Reconciliation #4089(watch) Kafka(kafka/kafka): Ignoring PersistentVolumeClaim data-0-kafka-kafka-0 diff {"op":"remove","path":"/metadata/finalizers"} 2022-04-12 23:21:09 DEBUG ResourceDiff:33 - Reconciliation #4089(watch) Kafka(kafka/kafka): Ignoring PersistentVolumeClaim data-0-kafka-kafka-0 diff {"op":"remove","path":"/metadata/managedFields"} 2022-04-12 23:21:09 DEBUG ResourceDiff:33 - Reconciliation #4089(watch) Kafka(kafka/kafka): Ignoring PersistentVolumeClaim data-0-kafka-kafka-0 diff {"op":"remove","path":"/status"} 2022-04-12 23:21:09 DEBUG AbstractResourceOperator:253 - Reconciliation #4089(watch) Kafka(kafka/kafka): PersistentVolumeClaim data-0-kafka-kafka-0 in namespace kafka did not changed and doesn't need patching 2022-04-12 23:21:09 DEBUG AbstractResourceOperator:115 - Reconciliation #4089(watch) Kafka(kafka/kafka): PersistentVolumeClaim kafka/data-0-kafka-kafka-1 already exists, patching it 2022-04-12 23:21:09 DEBUG ResourceDiff:33 - Reconciliation #4089(watch) Kafka(kafka/kafka): Ignoring PersistentVolumeClaim data-0-kafka-kafka-1 diff {"op":"remove","path":"/metadata/annotations/pv.kubernetes.io~1bind-completed"}

[scholzj]

It does the patching, but that updates things such as metadata. It does not change the data inside the volume.

[hari819]

@scholzj ,

i was able to observe that the zookeeper snapshot file has all the old data(topics) which should get deleted when i deleted all the PVCs in the kafka namespace , based on the topics data in the zookeeper snapshot file , kafka data was created where each index and log file in /var/lib/kafka/data was just an empty file(with each file of size 10mb) , all this data was accumulated to around 200GB in each broker ,

i am trying to find out how the zookeeper was able to restore the old snapshot file even after i deleted all PVCs in the namespace ,

[hari819]

we use moosefs as the file system in the kubernetes cluster ,https://moosefs.com/

MooseFS is a Fault-tolerant, Highly available, Highly performing, Scaling-out, Network distributed file system. It spreads data over several physical commodity servers, which are visible to the user as one virtual disk. It is POSIX compliant and acts like any other Unix-like file system supporting:

Hierarchical structure: Files and Folders,
File attributes,
Special files: Pipes, Sockets, Block and Character devices,
Symbolic and Hard links,
Security attributes and ACLs.
It works with all applications that require a standard file system.

[scholzj]

Never heard about that, so I don't know. I don't even know if it is a storage suitable for Kafka.

[hari819]

@scholzj , got the culprit who is storing the old topics even after the PVCs got deleted , it is this "Kafka Topics" CRD ,

nothing to do with any filesystem ,FS works perfectly for Kafka

[scholzj]

That can recreate the topics. But not the data as you suggested before.

[hari819]

@scholzj , the data in the topics is just an empty index file , log file with each having size of 10mb