Having trouble resizing #364

d2orbc · 2025-05-21T19:55:23Z

d2orbc
May 21, 2025

I tried resizing a PVC while it was mounted to a running pod. Here's the log:

csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767208       1 main.go:111] "Version" version="v1.13.1"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767305       1 feature_gate.go:387] feature gates: {map[]}
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767630       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767657       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767669       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.767675       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.768925       1 connection.go:234] "Connecting" address="unix:///csi/csi.sock"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.771270       1 common.go:143] "Probing CSI driver for readiness"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.771303       1 connection.go:264] "GRPC call" method="/csi.v1.Identity/Probe" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.774376       1 connection.go:270] "GRPC response" response="{\"ready\":{\"value\":true}}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.774409       1 connection.go:264] "GRPC call" method="/csi.v1.Identity/GetPluginInfo" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.775168       1 connection.go:270] "GRPC response" response="{\"name\":\"csi.proxmox.sinextra.dev\",\"vendor_version\":\"0.5.0\"}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.775203       1 main.go:166] "CSI driver name" driverName="csi.proxmox.sinextra.dev"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.775217       1 connection.go:264] "GRPC call" method="/csi.v1.Identity/GetPluginCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.775943       1 connection.go:270] "GRPC response" response="{\"capabilities\":[{\"service\":{\"type\":\"CONTROLLER_SERVICE\"}},{\"service\":{\"type\":\"VOLUME_ACCESSIBILITY_CONSTRAINTS\"}},{\"volume_expansion\":{\"type\":\"ONLINE\"}}]}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.775985       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerGetCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.776752       1 connection.go:270] "GRPC response" response="{\"capabilities\":[{\"rpc\":{\"type\":\"CREATE_DELETE_VOLUME\"}},{\"rpc\":{\"type\":\"PUBLISH_UNPUBLISH_VOLUME\"}},{\"rpc\":{\"type\":\"GET_CAPACITY\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}},{\"rpc\":{\"type\":\"MODIFY_VOLUME\"}}]}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.776788       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerGetCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.777186       1 connection.go:270] "GRPC response" response="{\"capabilities\":[{\"rpc\":{\"type\":\"CREATE_DELETE_VOLUME\"}},{\"rpc\":{\"type\":\"PUBLISH_UNPUBLISH_VOLUME\"}},{\"rpc\":{\"type\":\"GET_CAPACITY\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}},{\"rpc\":{\"type\":\"MODIFY_VOLUME\"}}]}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.777225       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerGetCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.777657       1 connection.go:270] "GRPC response" response="{\"capabilities\":[{\"rpc\":{\"type\":\"CREATE_DELETE_VOLUME\"}},{\"rpc\":{\"type\":\"PUBLISH_UNPUBLISH_VOLUME\"}},{\"rpc\":{\"type\":\"GET_CAPACITY\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}},{\"rpc\":{\"type\":\"MODIFY_VOLUME\"}}]}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.778437       1 leaderelection.go:257] attempting to acquire leader lease csi-proxmox/external-resizer-csi-proxmox-sinextra-dev...
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.791486       1 leaderelection.go:271] successfully acquired lease csi-proxmox/external-resizer-csi-proxmox-sinextra-dev
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.791623       1 leader_election.go:192] "New leader detected" leader="proxmox-csi-plugin-controller-5d5f956f4c-j4m9s"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.791890       1 leader_election.go:184] "became leader, starting"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.791926       1 controller.go:274] "Starting external resizer" controller="csi.proxmox.sinextra.dev"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.792051       1 reflector.go:313] Starting reflector *v1.PersistentVolumeClaim (10m0s) from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.792069       1 reflector.go:313] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.792083       1 reflector.go:349] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.792089       1 reflector.go:349] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.796567       1 reflector.go:376] Caches populated for *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.796714       1 reflector.go:376] Caches populated for *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:160
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.797064       1 leaderelection.go:293] successfully renewed lease csi-proxmox/external-resizer-csi-proxmox-sinextra-dev
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892213       1 controller.go:326] "Started PVC processing for resize controller" key="default/cluster-example-2"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892224       1 controller.go:326] "Started PVC processing for resize controller" key="default/data-volume-template-chi-simple-01-simple-0-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892236       1 controller.go:326] "Started PVC processing for resize controller" key="default/cluster-example-1"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892292       1 controller.go:374] "No need to resize PVC" PVC="default/data-volume-template-chi-simple-01-simple-0-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892310       1 controller.go:374] "No need to resize PVC" PVC="default/cluster-example-1"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892256       1 controller.go:374] "No need to resize PVC" PVC="default/cluster-example-2"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892336       1 controller.go:326] "Started PVC processing for resize controller" key="default/data-volume-template-chi-simple-01-simple-1-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892332       1 controller.go:326] "Started PVC processing for resize controller" key="default/log-volume-template-chi-simple-01-simple-0-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892367       1 controller.go:374] "No need to resize PVC" PVC="default/log-volume-template-chi-simple-01-simple-0-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892316       1 controller.go:326] "Started PVC processing for resize controller" key="default/data-volume-template-chi-simple-01-simple-0-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892380       1 controller.go:326] "Started PVC processing for resize controller" key="default/log-volume-template-chi-simple-01-simple-1-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892396       1 controller.go:374] "No need to resize PVC" PVC="default/log-volume-template-chi-simple-01-simple-1-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892396       1 controller.go:326] "Started PVC processing for resize controller" key="default/log-volume-template-chi-simple-01-simple-1-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892405       1 controller.go:326] "Started PVC processing for resize controller" key="default/registry-pvc"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892410       1 controller.go:374] "No need to resize PVC" PVC="default/log-volume-template-chi-simple-01-simple-1-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892376       1 controller.go:326] "Started PVC processing for resize controller" key="default/log-volume-template-chi-simple-01-simple-0-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892425       1 controller.go:326] "Started PVC processing for resize controller" key="default/storage-test-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892435       1 controller.go:374] "No need to resize PVC" PVC="default/log-volume-template-chi-simple-01-simple-0-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892368       1 controller.go:374] "No need to resize PVC" PVC="default/data-volume-template-chi-simple-01-simple-1-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892318       1 controller.go:326] "Started PVC processing for resize controller" key="default/data-volume-template-chi-simple-01-simple-1-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892472       1 controller.go:374] "No need to resize PVC" PVC="default/data-volume-template-chi-simple-01-simple-1-0-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892415       1 controller.go:374] "No need to resize PVC" PVC="default/registry-pvc"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.892382       1 controller.go:374] "No need to resize PVC" PVC="default/data-volume-template-chi-simple-01-simple-0-1-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.902392       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerGetCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.902520       1 event.go:389] "Event occurred" object="default/storage-test-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Resizing" message="External resizer is resizing volume pvc-ae04487b-b1e8-4b21-a042-88cd985bfcef"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.903111       1 connection.go:270] "GRPC response" response="{\"capabilities\":[{\"rpc\":{\"type\":\"CREATE_DELETE_VOLUME\"}},{\"rpc\":{\"type\":\"PUBLISH_UNPUBLISH_VOLUME\"}},{\"rpc\":{\"type\":\"GET_CAPACITY\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}},{\"rpc\":{\"type\":\"MODIFY_VOLUME\"}}]}" err=null
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:44.903154       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerExpandVolume" request="{\"capacity_range\":{\"required_bytes\":2147483648},\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"fs_type\":\"ext4\"}},\"volume_id\":\"cozy/ada1/local-zfs/vm-9999-pvc-ae04487b-b1e8-4b21-a042-88cd985bfcef\"}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:45.001824       1 connection.go:270] "GRPC response" response="{}" err="rpc error: code = Unavailable desc = error reading from server: EOF"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer E0521 19:52:45.001951       1 controller.go:316] "Error syncing PVC" err="resize volume \"pvc-ae04487b-b1e8-4b21-a042-88cd985bfcef\" by resizer \"csi.proxmox.sinextra.dev\" failed: rpc error: code = Unavailable desc = error reading from server: EOF"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:45.002310       1 event.go:389] "Event occurred" object="default/storage-test-0" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="VolumeResizeFailed" message="resize volume \"pvc-ae04487b-b1e8-4b21-a042-88cd985bfcef\" by resizer \"csi.proxmox.sinextra.dev\" failed: rpc error: code = Unavailable desc = error reading from server: EOF"
- csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s › proxmox-csi-plugin-controller
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:46.003090       1 controller.go:326] "Started PVC processing for resize controller" key="default/storage-test-0"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer I0521 19:52:46.011257       1 connection.go:264] "GRPC call" method="/csi.v1.Controller/ControllerGetCapabilities" request="{}"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer E0521 19:52:46.011399       1 connection.go:208] "Lost connection" address="unix:///csi/csi.sock"
csi-proxmox proxmox-csi-plugin-controller-5d5f956f4c-j4m9s csi-resizer E0521 19:52:46.011532       1 connection.go:116] "Lost connection to CSI driver, exiting"

YAML:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test
  namespace: default
  labels:
    app: alpine
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: test
  replicas: 1
  template:
    metadata:
      labels:
        app: alpine
    spec:
      terminationGracePeriodSeconds: 3
      containers:
        - name: alpine
          image: alpine
          command: ["sleep","1d"]
          securityContext:
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: storage
              mountPath: /mnt
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: alpine
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
        storageClassName: proxmox-data-local-zfs

Then I tried to kubectl edit pvc storage-test-0 to increase the size.

Answered by d2orbc

May 22, 2025

Fixed in #365

View full answer

d2orbc · 2025-05-22T17:29:51Z

d2orbc
May 22, 2025
Author

I0522 17:29:21.883819       1 controller.go:649] "ControllerExpandVolume: called" args="{\"capacity_range\":{\"required_bytes\":2147483648},\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"fs_type\":\"ext4\"}},\"volume_id\":\"cozy/ada1/local-zfs/vm-9999-pvc-ae04487b-b1e8-4b21-a042-88cd985bfcef\"}"
panic: interface conversion: interface {} is nil, not string

goroutine 58 [running]:
github.com/sergelogvinov/proxmox-csi-plugin/pkg/csi.(*ControllerService).ControllerExpandVolume(0xc000622200, {0x1ebfa40?, 0xa8000a?}, 0xc000120820)
	/src/pkg/csi/controller.go:719 +0x2337
github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerExpandVolume_Handler.func1({0x22a1718?, 0xc0002237a0?}, {0x1e34e40?, 0xc000120820?})
	/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi_grpc.pb.go:649 +0xcb
main.main.func1({0x22a1718?, 0xc0002237a0?}, {0x1e34e40?, 0xc000120820?}, 0x419845?, 0x80?)
	/src/cmd/controller/main.go:99 +0x3f
github.com/container-storage-interface/spec/lib/go/csi._Controller_ControllerExpandVolume_Handler({0x1ebfa40, 0xc000622200}, {0x22a1718, 0xc0002237a0}, 0xc0001d2400, 0x20cbef0)
	/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi_grpc.pb.go:651 +0x143
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001e6200, {0x22a1718, 0xc000223680}, 0xc00058fec0, 0xc0003035c0, 0x33bff68, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1405 +0x1036
google.golang.org/grpc.(*Server).handleStream(0xc0001e6200, {0x22a1e10, 0xc0000bcb60}, 0xc00058fec0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1815 +0xb88
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1035 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 54
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1046 +0x11d

I think this log statement is the culprit:

		if vmType, ok := vm["type"].(string); ok && vmType != "qemu" {
			klog.V(5).InfoS("ControllerExpandVolume: skipping non-qemu VM", "VM", vm["name"].(string)) //nolint:errcheck

			continue
		}

0 replies

d2orbc · 2025-05-22T18:07:30Z

d2orbc
May 22, 2025
Author

Fixed in #365

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Having trouble resizing #364

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Having trouble resizing #364

Uh oh!

d2orbc May 21, 2025

Replies: 2 comments

Uh oh!

Uh oh!

d2orbc May 22, 2025 Author

Uh oh!

d2orbc May 22, 2025 Author

d2orbc
May 21, 2025

d2orbc
May 22, 2025
Author

d2orbc
May 22, 2025
Author