Cant resize PVC

[veerendra2]

openshift v3.11.0+cbab8ee-94(K8s v1.11.0+d4cacc0)
Kafka Version: 2.6.0
Operators : 0.20.1

$ oc get storageclass generic -o yaml
allowVolumeExpansion: true   #<---------
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2019-02-04T09:31:05Z
  name: generic
  resourceVersion: "452378038"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/generic
  uid: 9837654e-285f-11e9-9f40-000d3ab3f50d
mountOptions:
- discard
parameters:
  kind: managed
  location: ${location}
  storageaccounttype: Premium_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

I followed, this blog -> https://strimzi.io/blog/2019/02/28/resizing-persistent-volumes/

kafka Cluster

---
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: sirius-dev
  namespace: kafka
spec:
  kafka:
    version: 2.6.0
    replicas: 3
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - sirius-dev-kafka
                topologyKey: "kubernetes.io/hostname"
    listeners:
      - name: tlsroute
        port: 9094
        type: route
        tls: true
        authentication:
          type: tls
    authorization:
      type: simple
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    config:
      offsets.topic.replication.factor: 2
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.6"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 256Gi
          deleteClaim: false
    metrics:
      # Inspired by config from Kafka 2.0.0 example rules:
      # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
      lowercaseOutputName: true
      rules:
        # Special cases and very specific rules
        - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
          name: kafka_server_$1_$2
          type: GAUGE
          labels:
            clientId: "$3"
            topic: "$4"
            partition: "$5"
        - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
          name: kafka_server_$1_$2
          type: GAUGE
          labels:
            clientId: "$3"
            broker: "$4:$5"
        - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
          name: kafka_server_$1_connections_tls_info
          type: GAUGE
          labels:
            listener: "$2"
            networkProcessor: "$3"
            protocol: "$4"
            cipher: "$5"
        - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
          name: kafka_server_$1_connections_software
          type: GAUGE
          labels:
            clientSoftwareName: "$2"
            clientSoftwareVersion: "$3"
            listener: "$4"
            networkProcessor: "$5"
        - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
          name: kafka_server_$1_$4
          type: GAUGE
          labels:
            listener: "$2"
            networkProcessor: "$3"
        - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
          name: kafka_server_$1_$4
          type: GAUGE
          labels:
            listener: "$2"
            networkProcessor: "$3"
        # Some percent metrics use MeanRate attribute
        # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
          name: kafka_$1_$2_$3_percent
          type: GAUGE
        # Generic gauges for percents
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
          name: kafka_$1_$2_$3_percent
          type: GAUGE
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
          name: kafka_$1_$2_$3_percent
          type: GAUGE
          labels:
            "$4": "$5"
        # Generic per-second counters with 0-2 key/value pairs
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
        # Generic gauges with 0-2 key/value pairs
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
        # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
        # Note that these are missing the '_sum' metric!
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            "$6": "$7"
            quantile: "0.$8"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            quantile: "0.$6"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            quantile: "0.$4"
  zookeeper:
    replicas: 3
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - sirius-dev-zookeeper
                topologyKey: "kubernetes.io/hostname"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: 64Gi
      deleteClaim: false
    metrics:
      # Inspired by Zookeeper rules
      # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/zookeeper.yaml
      lowercaseOutputName: true
      rules:
        # replicated Zookeeper
        - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
          name: "zookeeper_$2"
          type: GAUGE
        - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
          name: "zookeeper_$3"
          type: GAUGE
          labels:
            replicaId: "$2"
        - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
          name: "zookeeper_$4"
          type: COUNTER
          labels:
            replicaId: "$2"
            memberType: "$3"
        - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
          name: "zookeeper_$4"
          type: GAUGE
          labels:
            replicaId: "$2"
            memberType: "$3"
        - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
          name: "zookeeper_$4_$5"
          type: GAUGE
          labels:
            replicaId: "$2"
            memberType: "$3"
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

oc get pvc

$ oc get pvc
NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-0-sirius-dev-kafka-0     Bound     pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-0-sirius-dev-kafka-1     Bound     pvc-ae506870-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-0-sirius-dev-kafka-2     Bound     pvc-ae510d61-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-sirius-dev-zookeeper-0   Bound     pvc-6b6fd686-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d
data-sirius-dev-zookeeper-1   Bound     pvc-6b70ca79-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d
data-sirius-dev-zookeeper-2   Bound     pvc-6b706af2-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d

oc edit pvc data-0-sirius-dev-kafka-0

...
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 257Gi # <---- It was 256Gi, edited to 257Gi
  storageClassName: generic
  volumeName: pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d
...

I edited all PVCs, changed size to 257Gi. Waited for some time, checked the status

$ oc get pvc data-0-sirius-dev-kafka-0 -o yaml
...
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 257Gi
  storageClassName: generic
  volumeName: pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 256Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2021-10-22T14:31:05Z
    status: "True"
    type: Resizing  #<------ Status not changed!
  phase: Bound
...

But I did rolling update anyways

$ kubectl annotate statefulset sirius-dev-kafka strimzi.io/manual-rolling-update=true

Waited some time, didn't resized

$ oc get pvc
NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-0-sirius-dev-kafka-0     Bound     pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-0-sirius-dev-kafka-1     Bound     pvc-ae506870-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-0-sirius-dev-kafka-2     Bound     pvc-ae510d61-30d8-11ec-913f-000d3ab3f21d   256Gi      RWO            generic        3d
data-sirius-dev-zookeeper-0   Bound     pvc-6b6fd686-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d
data-sirius-dev-zookeeper-1   Bound     pvc-6b70ca79-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d
data-sirius-dev-zookeeper-2   Bound     pvc-6b706af2-30d8-11ec-913f-000d3ab3f21d   64Gi       RWO            generic        3d

Broker pod events

$ oc describe pod sirius-dev-kafka-0
...
Events:
  Type     Reason                  Age              From                     Message
  ----     ------                  ----             ----                     -------
  Normal   Scheduled               1h               default-scheduler        Successfully assigned kafka/sirius-dev-kafka-0 to ocp-node-dev-1
  Warning  FailedAttachVolume      1h               attachdetach-controller  Multi-Attach error for volume "pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d" Volume is already exclusively attached to one node and can't be attached to another
  Normal   SuccessfulAttachVolume  1h               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d"
  Warning  FailedMount             1h (x4 over 1h)  kubelet, ocp-node-dev-1  MountVolume.WaitForAttach failed for volume "pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d" : Cannot find Lun for disk kubernetes-dynamic-pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d
  Normal   Pulled                  1h               kubelet, ocp-node-dev-1  Container image "strimzi/kafka:0.20.1-kafka-2.6.0" already present on machine
  Normal   Created                 1h               kubelet, ocp-node-dev-1  Created container
  Normal   Started                 1h               kubelet, ocp-node-dev-1  Started container

PVC events

$ oc describe pvc data-0-sirius-dev-kafka-0
...
Events:
  Type     Reason              Age               From           Message
  ----     ------              ----              ----           -------
  Warning  VolumeResizeFailed  1h (x4 over 1h)   volume_expand  Error expanding volume "kafka/data-0-sirius-dev-kafka-0" of plugin kubernetes.io/azure-disk : compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d while it is attached to running VM /subscriptions/fd68ac/resourceGroups/sirius-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-5. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation."
  Warning  VolumeResizeFailed  3m (x22 over 1h)  volume_expand  Error expanding volume "kafka/data-0-sirius-dev-kafka-0" of plugin kubernetes.io/azure-disk : compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d while it is attached to running VM /subscriptions/fdresourceGroups/sirius-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-1. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation."

I have also asked question in stackoverflow: https://stackoverflow.com/questions/69519063/strimzi-cant-resize-pv

[scholzj]

Cannot resize disk kubernetes-dynamic-pvc-ae5046bb-30d8-11ec-913f-000d3ab3f21d while it is attached to running VM /subscriptions/fd667dc8-1e9e-4ab4-b428-9879d458e8ac/resourceGroups/sirius-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-5. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation.

Well, in the error the infrastructure is telling you what the error is and it does not seem to have much do with Strimzi. At the end, you are resizing the PVCs => so you are not even using the Strimzi feature for resizing the volumes. I guess you will need to follow what it suggests and detach the volumes?

[veerendra2]

@scholzj

At the end, you are resizing the PVCs => so you are not even using the Strimzi feature for resizing the volumes

Sorry what is Strimzi feature? or did I miss any?. Or some thing related to Azure,Azure/AKS#1477 Have scale down 0 to shutoff the pod and then resize happens.

Or is there any why temporarily shutoff the cluster like maintenance or something

[scholzj]

The proper way to resize the storage these days with Strimzi is to change it in the Kafka custom resource and let Strimzi and Kubernetes do it. But TBH you have to be using quite old version since you use Kubernetes 1.11 - so I'm not sure if this is supported in your version. There is also no guarantee that it does not end up in the same error as it ended here. The point I was trying to make is that Strimzi was not involved at all in the steps you did. It was only your Kubernetes / OpenShift and your infrastructure. So they might be better suited to help you with the error.

[veerendra2]

Thanks for the quick reply. I have changed Kafka resource's volume size. But still same issue

Events:
  Type     Reason                 Age              From                         Message
  ----     ------                 ----             ----                         -------
  Normal   ProvisioningSucceeded  55m              persistentvolume-controller  Successfully provisioned volume pvc-8b2427f0-357a-11ec-9048-000d3ab3f21d using kubernetes.io/azure-disk
  Warning  VolumeResizeFailed     1m (x5 over 8m)  volume_expand                Error expanding volume "kafka/data-0-sirius-dev-kafka-0" of plugin kubernetes.io/azure-disk : compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-8b2427f0-357a-11ec--000d3ab3f21d while it is attached to running VM /subscriptions/fd667dc8--4ab4-b428/resourceGroups/sirius-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-5. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation."

I guess because of old cluster like you mentioned

[orovp]

This situation still exist on Openshift 4.10.28, (k8 1.23).
My steps to resize are:

• Me: Update Kafka resource with new storage size
• Strimzi automatically update PVC specs
• Storage driver update volume -> This step only can be completed with a volume detaching
• Statefulset rollup triggered by Strimzi

On my cluster, Strimzi trigger last step (rollup), but Storage driver doesn't have enough time to resize the disk during rollup.

Is it possible to make a check on Strimzi during resize operation to wait untill pvc is mount? You could use FileSystemResizePending on PVC spec:

jsonpath="{.status.conditions[?(@.type=='FileSystemResizePending')]}

Something like:

• Strimzi update PVC spec
• Strimzi Stop Kafka pod
• Decide when to start pod using FileSystemResizePending data, and only start pod if volume is resized

Thank you,
Oscar

[scholzj]

I'm not sure I follow what are you asking for. Strimzi does check for the FileSystemResizePending and roll the pods when it is set.

[orovp]

Yes, but I guess a roll step is an async operation that doesn't wait resize operation from storage driver. As you say, roll is done, but storage driver fail during first trigger to change size of volume because attached, and future retries don't fit roll window when process is down.

On my case, after Kafka spec is changed with a new storage size, pvc is updated and new condition entry is filled:

conditions:
    - type: Resizing
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-03-21T16:09:00Z'

Strimzi make a roll that restart pod, but storage driver has not resized volume because trigger of resize was made out of roll window (when process is down). In my opinion, a roll action is not apropiate on this situation, operator should stop pod and wait until volume is resized by storage driver. (Maybe is donde by strimzi, but following my experience of last days it seems)

An expected situation from my point of view observing my cluster reaction, following steps should be executed on a sync way, where next step only can start if previous steps has finished:

1. Kafka spec is changed with new storage size
2. Strimzi update PVC and condition with type Resizing is filled into PVC spec.
3. Kafka Statefulset roll has begun and pod is down
4. Storage Driver resize volume and condition with type Resizing is changed by type FileSystemResizePending
5. Kafka Statefulset roll start pod
6. PVC spec in status node is increased with Bound phase

My current situation is that step 5 is donde before step 4 has finished. Where steps 1,2 and 3 are done and final state of PVC is with Resizing condition type.

FileSystemResizePending state is reached when I manually scale down statefulset and storage driver make its job. I didn't explain me well on previous post.

As I understand, step 3 and step 4 are not syncronized because storage driver and roll process are independent, but I guess that should be sync on some way, maybe could be enough that strimzi stop when PVC Resizing condition is reached and wait until FileSystemResizePending=true with message "Waiting for user to (re-)start a pod to finish file system resize of
volume on node" to start pod.

Could you explain how strimzi take care of steps 3,4 and 5? How does strimzi know that should start step 5?
It seems that Strimzi send a rollout restart on statefulset and leave this responsability to statefulset controller causing a race condition between controller and storage driver.

Thank you for fast response.

[scholzj]

I think you are misunderstanding what is happening and why. If you still use StatefulSets as you suggest because of using old version of Strimzi or because of disabling StrimziPodSets, then the first rolling is because of the configuration change itself and has nothing to do with the state of the PVCs or with waiting for them.

The actual resizing itself is quite complicated. And I'm afraid it is not very well standardized and works differently across platforms. Strimzi requests the resizing in the PVC. But it can happen only in a few hours for example. So no, we cannot shut down a pod and wait for it to happen. So Strimzi requests the resizing by changing the PVC and goes with its work as usual. It periodically checks if the PVC status is in FileSystemResizePending. That typically indicates that the pod needs to be restarted to finish the resizing. So when Strimzi sees it, it rolls the corresponding pod.

However, there are many traps in this:

• Some platforms use the FileSystemResizePending condition but do not actually need the pod to restart. They just indicate that the resizing is happening but it happens online (this happens for example on AWS) => in this case we might restart the pod unnecessarily, but ok.
• Some platforms use the FileSystemResizePending condition but restarting the pod is not enough since they need it to be unmounted completely from the worker node and Strimzi itself cannot do that. In that case, you might need to handle it manually (this happens for example on Azure).
• Other platforms do something completely different and for example use other conditions to indicate what they need. We unfortunatelly have no change to implement and especially test custom code for every single one of them.

So a lot on how it actually works or does not work depends on what Kubernetes distribution you run and on which infrastructure.

[orovp]

Thank you @scholzj, now is much clearer to me. My provider is Azure.

About how to manually manage this resize operation, I've seen on internet recommendation about delete statefulset instead to scale down.

On my case it seems a scale down is enough but could you give me some recommendation or best practice about that? or maybe give me some documentation to review?

When I say "scale down" I mean that I scale down kafka and strimzi cluster operator.

Thank you.

[scholzj]

TBH, I do not really have any recommendation for that and I do not really use Azure myself. So not sure how exactly it works. I guess you can stop the operator, do a non-cascading delete of the StatefulSet and then delete the pod you need => that would be what I would do.