hpe-nfs volume mounts fail after upgrade from OpenShift 4.18 to 4.20 #483
Description
I asked my openshift cluster to update over the weekend. When I got in on Monday the upgrade was in a locked state as the the nodes where unable to evict the hpe-nfs pods. Also, the Nimble had hit the 1024 volume maximum as some pod had crash looped all weekend creating a new volume on every crash.
As I needed the nodes to finish the update I manually deleted the troubling pods. OCP is up and running, but the pods in hpe-nfs are unable to function. I also deleted all offline volumes on the Nimble.
I've deleted an re-created the HPECSIDriver - but things remain broken. Unsure how to troubleshoot properly.
$ oc events -n testns -w
LAST SEEN TYPE REASON OBJECT MESSAGE
3m39s (x64 over 18m) Normal ExternalProvisioning PersistentVolumeClaim/asd Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
113s (x13 over 18m) Normal Provisioning PersistentVolumeClaim/asd External provisioner is provisioning volume for claim "testns/asd"
3m40s (x3 over 18m) Warning ProvisioningFailed PersistentVolumeClaim/asd failed to provision volume with StorageClass "hpe-standard": rpc error: code = DeadlineExceeded desc = context deadline exceeded
113s (x10 over 18m) Warning ProvisioningFailed PersistentVolumeClaim/asd failed to provision volume with StorageClass "hpe-standard": rpc error: code = Aborted desc = There is already an operation pending for the specified id CreateVolume:pvc-06309287-ccfb-4126-82e4-803624a910d0
4m34s (x2 over 13m) Warning ProvisionStorage PersistentVolumeClaim/asd gave up waiting for deployment hpe-nfs-06309287-ccfb-4126-82e4-803624a910d0 to be available
5h10m (x965 over 2d21h) Normal Provisioning PersistentVolumeClaim/testnss-first External provisioner is provisioning volume for claim "testns/testnss-first"
5h15m (x699 over 2d21h) Warning ProvisioningFailed PersistentVolumeClaim/testnss-first failed to provision volume with StorageClass "hpe-standard": rpc error: code = DeadlineExceeded desc = context deadline exceeded
5h33m (x263 over 2d21h) Warning ProvisioningFailed PersistentVolumeClaim/testnss-first failed to provision volume with StorageClass "hpe-standard": rpc error: code = Aborted desc = There is already an operation pending for the specified id CreateVolume:pvc-3bd02ce0-1b43-4acd-bce2-4d9f14c5aae3
5h11m (x697 over 2d21h) Warning ProvisionStorage PersistentVolumeClaim/testnss-first gave up waiting for deployment hpe-nfs-3bd02ce0-1b43-4acd-bce2-4d9f14c5aae3 to be available
5h15m (x15683 over 2d21h) Normal ExternalProvisioning PersistentVolumeClaim/testnss-first Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
5h22m (x2 over 5h27m) Warning ProvisionStorage PersistentVolumeClaim/testnss-first gave up waiting for pvc hpe-nfs-3bd02ce0-1b43-4acd-bce2-4d9f14c5aae3 to be bound
5h10m Warning ProvisioningFailed PersistentVolumeClaim/testnss-first failed to provision volume with StorageClass "hpe-standard": rpc error: code = Internal desc = Failed to configure create parameters from PVC annotations: Requested pvc pvc-3bd02ce0-1b43-4acd-bce2-4d9f14c5aae3 was not found with uid 3bd02ce0-1b43-4acd-bce2-4d9f14c5aae3
9s (x8 over 100s) Normal ExternalProvisioning PersistentVolumeClaim/rwo Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
7s (x7 over 100s) Normal Provisioning PersistentVolumeClaim/rwo External provisioner is provisioning volume for claim "testns/rwo"
70s Warning ProvisioningFailed PersistentVolumeClaim/rwo failed to provision volume with StorageClass "hpe-standard": rpc error: code = DeadlineExceeded desc = context deadline exceeded
7s (x6 over 69s) Warning ProvisioningFailed PersistentVolumeClaim/rwo failed to provision volume with StorageClass "hpe-standard": rpc error: code = Aborted desc = There is already an operation pending for the specified id CreateVolume:pvc-71066741-ee8d-4390-a2c0-ef5cd4041817
147m (x10 over 154m) Normal Provisioning PersistentVolumeClaim/test External provisioner is provisioning volume for claim "testns/test"
152m (x11 over 154m) Normal ExternalProvisioning PersistentVolumeClaim/test Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
153m Warning ProvisioningFailed PersistentVolumeClaim/test failed to provision volume with StorageClass "hpe-standard": rpc error: code = DeadlineExceeded desc = context deadline exceeded
151m (x8 over 153m) Warning ProvisioningFailed PersistentVolumeClaim/test failed to provision volume with StorageClass "hpe-standard": rpc error: code = Aborted desc = There is already an operation pending for the specified id CreateVolume:pvc-79889285-fee0-4cf8-bbc8-2f2764e93312
149m Warning ProvisionStorage PersistentVolumeClaim/test gave up waiting for deployment hpe-nfs-79889285-fee0-4cf8-bbc8-2f2764e93312 to be available
147m Warning ProvisioningFailed PersistentVolumeClaim/test failed to provision volume with StorageClass "hpe-standard": rpc error: code = Internal desc = Failed to configure create parameters from PVC annotations: Requested pvc pvc-79889285-fee0-4cf8-bbc8-2f2764e93312 was not found with uid 79889285-fee0-4cf8-bbc8-2f2764e93312