"Defined disk interface not supported" when applying VM changes?

[kreeger]

Here's the setup.

• I'm using sergelogvinov/proxmox-csi-plugin for provisioning and attaching Proxmox disks to Proxmox guests for Kubernetes workloads.
• With existing workloads attached and ignore_changes = [disk] in my resource's lifecycle block, when I apply changes to VMs that have additional disks attached, I get the following error for each VM.

Error: Defined disk interface not supported. Interface was , but only [ide sata scsi virtio] are supported

    with module.workers.proxmox_virtual_environment_vm.this["worker-199c47f9"],
    on ../proxmox-vms/main.tf line 1, in resource "proxmox_virtual_environment_vm" "this":
    1: resource "proxmox_virtual_environment_vm" "this" {

Here's what the state for this VM looks like, in particular.

Click to expand JSON state for `worker-199c47f9`

{
  "index_key": "worker-199c47f9",
  "schema_version": 0,
  "attributes": {
    "acpi": true,
    "agent": [
      {
        "enabled": true,
        "timeout": "15m",
        "trim": false,
        "type": "virtio"
      }
    ],
    "amd_sev": [],
    "audio_device": [],
    "bios": "seabios",
    "boot_order": null,
    "cdrom": [],
    "clone": [],
    "cpu": [
      {
        "affinity": "",
        "architecture": "",
        "cores": 8,
        "flags": [],
        "hotplugged": 0,
        "limit": 0,
        "numa": false,
        "sockets": 1,
        "type": "x86-64-v2-AES",
        "units": 1024
      }
    ],
    "description": "Managed by Terraform",
    "disk": [
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi3",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-ab62b9d5-3003-4bb8-9f3b-cfd6b51f7384",
        "replicate": true,
        "serial": "",
        "size": 1,
        "speed": [],
        "ssd": false
      },
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi1",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-c6e57fdb-9e6a-4af4-8848-a093396b7ba4",
        "replicate": true,
        "serial": "",
        "size": 1,
        "speed": [],
        "ssd": false
      },
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi6",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-6a764f3f-997e-469e-8dd0-83ee8d11b282",
        "replicate": true,
        "serial": "",
        "size": 1,
        "speed": [],
        "ssd": false
      },
      {
        "aio": null,
        "backup": null,
        "cache": null,
        "datastore_id": null,
        "discard": null,
        "file_format": "",
        "file_id": "",
        "import_from": "",
        "interface": "",
        "iothread": null,
        "path_in_datastore": "",
        "replicate": null,
        "serial": "",
        "size": null,
        "speed": [],
        "ssd": null
      },
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi5",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-c93aba6b-e111-4b0d-b199-6ec39c1875ec",
        "replicate": true,
        "serial": "",
        "size": 10,
        "speed": [],
        "ssd": false
      },
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi2",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-55c2ab82-3e40-4a5b-a11c-e1115f743997",
        "replicate": true,
        "serial": "",
        "size": 32,
        "speed": [],
        "ssd": false
      },
      {
        "aio": "io_uring",
        "backup": true,
        "cache": "none",
        "datastore_id": "vms-x64",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "virtio0",
        "iothread": true,
        "path_in_datastore": "vm-513-disk-0",
        "replicate": true,
        "serial": "",
        "size": 96,
        "speed": [],
        "ssd": false
      },
      {
        "aio": "io_uring",
        "backup": false,
        "cache": "none",
        "datastore_id": "k8s",
        "discard": "ignore",
        "file_format": "raw",
        "file_id": "",
        "import_from": "",
        "interface": "scsi4",
        "iothread": true,
        "path_in_datastore": "vm-9999-pvc-d489273b-b3f2-4c10-ab2d-9b8c2d004068",
        "replicate": true,
        "serial": "",
        "size": 1,
        "speed": [],
        "ssd": false
      }
    ],
    "efi_disk": [],
    "hook_script_file_id": null,
    "hostpci": [],
    "id": "513",
    "initialization": [
      {
        "datastore_id": "vms-x64",
        "dns": [],
        "interface": "ide2",
        "ip_config": [
          {
            "ipv4": [
              {
                "address": "192.168.100.22/24",
                "gateway": "192.168.100.1"
              }
            ],
            "ipv6": []
          }
        ],
        "meta_data_file_id": "",
        "network_data_file_id": "",
        "type": "",
        "user_account": [],
        "user_data_file_id": "",
        "vendor_data_file_id": ""
      }
    ],
    "ipv4_addresses": [
      [
        "127.0.0.1",
        "169.254.116.108"
      ],
      [],
      [],
      [],
      [],
      [],
      [],
      [
        "192.168.100.22"
      ],
      [],
      [
        "10.244.2.163"
      ],
      [],
      [],
      [],
      []
    ],
    "ipv6_addresses": [
      [
        "::1"
      ],
      [],
      [],
      [],
      [],
      [],
      [],
      [
        "fe80::be24:11ff:feb3:ce42"
      ],
      [
        "fe80::34ca:29ff:fee7:1ce3"
      ],
      [
        "fe80::2878:ebff:fe28:40de"
      ],
      [
        "fe80::84e5:b3ff:fe21:bfcc"
      ],
      [
        "fe80::84f:35ff:fefc:6c6f"
      ],
      [
        "fe80::48be:a3ff:fe53:7ccd"
      ],
      [
        "fe80::a0cf:3ff:fef7:2aa9"
      ]
    ],
    "keyboard_layout": "en-us",
    "kvm_arguments": "",
    "mac_addresses": [
      "00:00:00:00:00:00",
      "DE:4C:5D:34:EA:18",
      "32:F7:2C:96:4A:2B",
      "00:00:00:00:00:00",
      "00:00:00:00:00:00",
      "00:00:00:00:00:00",
      "00:00:00:00:00:00",
      "BC:24:11:B3:CE:42",
      "36:CA:29:E7:1C:E3",
      "2A:78:EB:28:40:DE",
      "86:E5:B3:21:BF:CC",
      "0A:4F:35:FC:6C:6F",
      "4A:BE:A3:53:7C:CD",
      "A2:CF:03:F7:2A:A9"
    ],
    "machine": "",
    "memory": [
      {
        "dedicated": 16384,
        "floating": 0,
        "hugepages": "",
        "keep_hugepages": false,
        "shared": 0
      }
    ],
    "migrate": false,
    "name": "worker-199c47f9",
    "network_device": [
      {
        "bridge": "vmbr10",
        "disconnected": false,
        "enabled": true,
        "firewall": false,
        "mac_address": "BC:24:11:B3:CE:42",
        "model": "virtio",
        "mtu": 1,
        "queues": 0,
        "rate_limit": 0,
        "trunks": "",
        "vlan_id": 100
      }
    ],
    "network_interface_names": [
      "lo",
      "bond0",
      "dummy0",
      "teql0",
      "tunl0",
      "sit0",
      "ip6tnl0",
      "eth0",
      "cilium_net",
      "cilium_host",
      "cilium_vxlan",
      "lxc_health",
      "lxc17cb72a6ebbf",
      "lxcacc3b05ab550"
    ],
    "node_name": "proxmox1",
    "numa": [],
    "on_boot": true,
    "operating_system": [
      {
        "type": "l26"
      }
    ],
    "pool_id": null,
    "protection": false,
    "reboot": false,
    "reboot_after_update": true,
    "rng": [],
    "scsi_hardware": "virtio-scsi-pci",
    "serial_device": [
      {
        "device": "socket"
      }
    ],
    "smbios": [],
    "started": true,
    "startup": [],
    "stop_on_destroy": false,
    "tablet_device": true,
    "tags": [
      "terraform"
    ],
    "template": false,
    "timeout_clone": 1800,
    "timeout_create": 1800,
    "timeout_migrate": 1800,
    "timeout_move_disk": 1800,
    "timeout_reboot": 1800,
    "timeout_shutdown_vm": 1800,
    "timeout_start_vm": 1800,
    "timeout_stop_vm": 300,
    "tpm_state": [],
    "usb": [],
    "vga": [],
    "virtiofs": [],
    "vm_id": 513,
    "watchdog": []
  },
  "sensitive_attributes": [],
  "identity_schema_version": 0,
  "private": "bnVsbA=="
}

(I should note that this does not happen for nodes that do not have kinds of these disks attached.)

The disks are all scsiX, so how is the interface for these disks getting picked up as "" and not "scsi"? And why are these disks even tripping things up if I'm ignoring them in the resource's lifecycle?

---

Since I know just enough Go to get myself into trouble, I'll start poking around and see if this worth filing a bug report for. I'm on 0.80 of the provider.

• diskInterface is getting read as a blank string in disk.go here

[kreeger]

Ope. I see it in the .tfstate file in S3. For some reason there's a weird blank-ish disk in each VM's array of these disks.

{
  "aio": "io_uring",
  "backup": true,
  "cache": "none",
  "datastore_id": "local-lvm",
  "discard": "ignore",
  "file_format": "",
  "file_id": "",
  "import_from": "",
  "interface": "",
  "iothread": false,
  "path_in_datastore": "",
  "replicate": true,
  "serial": "",
  "size": 8,
  "speed": [],
  "ssd": false
},

So how does this get read into state in the first place? I'm not seeing it in my VM's qemu .conf file, and yet even when I do a terraform refresh, it's there in the state file.

Click to expand 513.conf

r# cat 513.conf
#Managed by Terraform
acpi: 1
agent: enabled=1,fstrim_cloned_disks=0,type=virtio
balloon: 0
bios: seabios
boot: order=virtio0;net0
cores: 8
cpu: cputype=x86-64-v2-AES
cpuunits: 1024
ide2: vms-x64:vm-513-cloudinit,media=cdrom
ipconfig0: gw=192.168.100.1,ip=192.168.100.22/24
keyboard: en-us
memory: 16384
meta: creation-qemu=9.2.0,ctime=1752597541
name: worker-199c47f9
net0: virtio=BC:24:11:B3:CE:42,bridge=vmbr10,firewall=0,mtu=1,tag=100
numa: 0
onboot: 1
ostype: l26
protection: 0
scsi1: k8s:vm-9999-pvc-c6e57fdb-9e6a-4af4-8848-a093396b7ba4,backup=0,iothread=1,size=1G,wwn=0x5056432d49443031
scsi10: k8s:vm-9999-pvc-0f11cd9e-a639-4165-adad-63385c0dfedc,backup=0,iothread=1,size=5G,wwn=0x5056432d49443130
scsi2: k8s:vm-9999-pvc-55c2ab82-3e40-4a5b-a11c-e1115f743997,backup=0,iothread=1,size=32G,wwn=0x5056432d49443032
scsi3: k8s:vm-9999-pvc-ab62b9d5-3003-4bb8-9f3b-cfd6b51f7384,backup=0,iothread=1,size=1G,wwn=0x5056432d49443033
scsi4: k8s:vm-9999-pvc-d489273b-b3f2-4c10-ab2d-9b8c2d004068,backup=0,iothread=1,size=1G,wwn=0x5056432d49443034
scsi5: k8s:vm-9999-pvc-c93aba6b-e111-4b0d-b199-6ec39c1875ec,backup=0,iothread=1,size=10G,wwn=0x5056432d49443035
scsi6: k8s:vm-9999-pvc-6a764f3f-997e-469e-8dd0-83ee8d11b282,backup=0,iothread=1,size=1G,wwn=0x5056432d49443036
scsi7: k8s:vm-9999-pvc-04f256ad-f7a5-46aa-b8f3-08b59d2ec42e,backup=0,iothread=1,size=1G,wwn=0x5056432d49443037
scsi8: k8s:vm-9999-pvc-0e73378e-05b6-4592-81c0-d46d1b1fad33,backup=0,iothread=1,size=1G,wwn=0x5056432d49443038
scsi9: k8s:vm-9999-pvc-4797b253-5966-47ee-8038-029c40d9ad01,backup=0,iothread=1,size=1G,wwn=0x5056432d49443039
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=fe3fef30-5e0b-4bfa-ae6f-7fd5f275510f
sockets: 1
tablet: 1
tags: terraform
template: 0
virtio0: vms-x64:vm-513-disk-0,aio=io_uring,backup=1,cache=none,discard=ignore,iothread=1,replicate=1,size=96G
vmgenid: 9f66f179-3b8d-453e-8d16-30204429ae34

If I edit the .tfstate file to remove those weird blank disks and then run … apply, that works.

[bpg]

Hey @kreeger 👋🏼

Your use case is probably not the best for terraform, as CSI driver is constantly updating the VM outside of the terraform, so there is always a drift in the resource that provider needs to reconcile on each run.
There are few limitations of the current provider that doesn't allow it to reconcile VM resource efficiently: disks represented as a list, not as a map with the interface as a key; legacy terraform SDK does not handle defaults well, etc. So, depends on the workflow, read / refresh / import operations may produce an inconsistent state.

[kreeger]

I'm good with that. I still plan to keep this under Terraform with proxmox-csi since it's just a hobby project anyway (and I will probably switch to a different storage driver at some point in the future), but your explanation helps shed some light!