Ubuntu 24.04 Cloud Image fails to provision on LVM storage while Ubuntu 22.04 works

[rodfranck]

Environment

• Terraform Provider: bpg/proxmox v0.95.0
• Proxmox Version: 9.1.4
• Node: node1
• Storage Type: LVM (mounted storage, appears as proxmox_storage_2)
• Network: Configured

Problem Description

I'm experiencing an issue where Ubuntu 22.04 cloud images deploy successfully to my LVM storage (proxmox_storage_2), but Ubuntu 24.04 cloud images fail with the exact same Terraform configuration.

What Works

• ✅ Ubuntu 22.04 cloud image (ubuntu-22.04-server-cloudimg-amd64.img) → LVM storage
• ✅ Ubuntu 24.04 cloud image → local-lvm storage
• ✅ Ubuntu 24.04 cloud image → NFS storage

What Doesn't Work

• ❌ Ubuntu 24.04 cloud image (ubuntu-24.04.3-server-cloudimg-amd64.img) → LVM storage (proxmox_storage_2)

The VM creation appears to complete, but the VM either:

• Fails to boot properly
• Doesn't initialize cloud-init correctly
• Has disk-related issues during first boot

Terraform Configuration

main.tf

resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
  name      = "test-ubuntu"
  node_name = "node1"
  stop_on_destroy = true

  agent {
    enabled = true
  }

  cpu {
    cores = 2
  }

  memory {
    dedicated = 2048
  }

  disk {
    datastore_id = "proxmox_storage_2"  # LVM storage - works with 22.04, fails with 24.04
    interface    = "scsi0"
    file_id      = "nfs-isos:iso/ubuntu-24.04.3-server-cloudimg-amd64.img"
    discard      = "on"
    size         = 10
  }

  initialization {
    dns {
      servers = ["10.4.62.102", "10.4.4.143"]
    }

    ip_config {
      ipv4 {
        address = "10.4.50.131/24"
        gateway = "10.4.50.254"
      }
    }

    user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
  }

  network_device {
    bridge = "vmbr70"
  }
}

provider.tf

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.95.0"
    }
  }
}

provider "proxmox" {
  endpoint  = "https://10.4.21.100:8006"
  username  = "root@pam"
  password  = "xxxxx"
  insecure  = true  

  ssh {
    agent    = true
    username = "root"

    node {
      name    = "node1"
      address = "10.4.21.100"
    }
  }
}

cloud-config.tf

resource "proxmox_virtual_environment_file" "user_data_cloud_config" {
  content_type = "snippets"
  datastore_id = "local"
  node_name    = "node1"

  source_raw {
    data = <<-EOF
    #cloud-config
    package_update: true
    timezone: America/Bogota
    hostname: test_bgp
    manage_etc_hosts: true

    users:
    - name: provision
      gecos: provision
      shell: /bin/bash
      sudo: ALL=(ALL) NOPASSWD:ALL
      lock_passwd: false
      passwd: "$6$hashed_password"

    packages:
    - vim
    - net-tools
    - qemu-guest-agent
    - htop

    runcmd:
    - systemctl enable --now qemu-guest-agent
    - sysctl --system

    final_message: "The system is finally up, after $UPTIME seconds"

    power_state:
      mode: reboot
      message: Restarting server after setup
    EOF

    file_name = "user-data-cloud-config.yaml"
  }
}

What I've Tried

1. Adding explicit cloud-init configuration:

   initialization {
     datastore_id = "proxmox_storage_2"
     interface    = "scsi1"
     # ... rest of config
   }

2. Changing SCSI controller:

   scsi_hardware = "virtio-scsi-single"

3. Enabling iothread:

   disk {
     # ...
     iothread = true
   }

4. Switching to virtio interface:

   disk {
     interface = "virtio0"
     # ...
   }

5. Switching to ovmf bios and efidisk:

   machine  = "q35"
   bios  = "ovmf"
   efi_disk {
       datastore_id = "local-lvm"
       type         = "4m"
   }

6. Add explicit boot order:

   boot_order = ["scsi0", "scsi1"]

7. Using template-based cloning (manually imported Ubuntu 24.04 to LVM via qm importdisk)

8. Different CPU types: x86-64-v2-AES, host, default

None of these solutions worked. The issue persists specifically with Ubuntu 24.04 + LVM storage combination.

Storage Information

pvesm status | grep proxmox_storage_2
proxmox_storage_2         lvm     active      4394524672       943722496      3450802176   21.47%

PVE Information

pveversion --verbose
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.4 (running version: 9.1.4/5ac30304265fbd8e)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17: 6.17.4-2
proxmox-kernel-6.17.2-1-pve-signed: 6.17.2-1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
dnsmasq: 2.91-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx11
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.1.4
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.4
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.0
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-3
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.1-1
proxmox-backup-file-restore: 4.1.1-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.5
pve-cluster: 9.0.7
pve-container: 6.0.18
pve-docs: 9.1.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.1.0
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-5
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.3
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1

[bpg-dev]

Hey @rodfranck 👋🏼

Have you checked https://github.com/bpg/terraform-provider-proxmox?tab=readme-ov-file#serial-device-required-for-debian-12--ubuntu-vms?

[bpg-dev]

Also, can you deploy the same configuration, but manually from the PVE UI?
If that works then it worth exporting vm config (/etc/pve/qemu-server/.conf) form UI-created VM and provider's created VM and compare them one by one.

[rodfranck]

Thanks for your answer @bpg-dev,

The issue is actually related to the LVM storage. I’m able to provision virtual machines using the Ubuntu 24 image with Cloud-Init as long as the disk storage (scsi0) is not set to the LVM storage (proxmox_storage_2). I can use NFS or Local-LVM without any problems.

I applied the required serial device change, but it still doesn’t work. I believe that change was not particularly relevant to the root cause.

Here is the configuration of the VM created using Local-LVM storage, which works correctly. However, if I switch the storage to proxmox_storage_2 (LVM), the VM fails during boot, indicating that it was unable to execute the Cloud-Init configuration.

#VM test 1
acpi: 1
agent: enabled=1,fstrim_cloned_disks=0,type=virtio
balloon: 0
bios: ovmf
boot: order=scsi0;scsi1
cicustom: user=local:snippets/user-data-10.4.70.130-Test-VM.yml
cores: 4
cpu: qemu64
efidisk0: local-lvm:vm-248-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M
ipconfig0: gw=10.4.70.254,ip=10.4.70.130/24
keyboard: en-us
machine: q35
memory: 4048
meta: creation-qemu=10.1.2,ctime=1770996645
name: 10.4.70.130-Test-VM
nameserver: 10.4.4.14 10.4.82.102
net0: virtio=BC:24:11:C2:1B:B0,bridge=vlan70,firewall=0
numa: 0
onboot: 0
ostype: l26
protection: 0
scsi0: local-lvm:vm-248-disk-1,aio=io_uring,backup=1,cache=none,discard=on,iothread=0,replicate=1,size=50G,ssd=1
scsi1: local-lvm:vm-248-cloudinit,media=cdrom
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=01087e86-ecdb-4200-b8f3-94e95d81a8ab
sockets: 1
tablet: 1
tags: gs;services;test
template: 0
vmgenid: d7358a05-8d82-48f8-8840-6f5fed8662e1

[rodfranck]

Yes, your assumption is correct — I was not able to deploy the image through the UI either; it failed when using the cloud ISO.

Regarding the discard=on and ssd=true parameters, I have also tried enabling and disabling them without success. The same applies to using the image via import_from.

Explicitly setting file_format = "raw" did not resolve the issue either.

At this point, I’ve basically spent two weeks permuting values and configurations to see if any combination magically works…

The same error also occurs when using the Telmate provider, so I’m increasingly convinced this is something related specifically to Ubuntu 24 rather than Terraform or the provider itself.

[rodfranck]

UPDATE:

After extensive testing, I've identified the actual problem:

The snippet file is created successfully by the provider, but Proxmox doesn't properly register/index it for use in VM creation within the same Terraform apply cycle.

Evidence

• If the snippet exists BEFORE running terraform apply, Ubuntu 24.04 works perfectly on LVM storage
• If Terraform creates the snippet and VM in the same apply, the VM fails to provision correctly (Ubuntu 24.04 only)
• Ubuntu 22.04 works either way

The proxmox_virtual_environment_file resource successfully uploads the snippet to the datastore, but there's a synchronization issue between:

1. The snippet being uploaded
2. Proxmox indexing/registering it in the storage
3. The VM being able to reference it via user_data_file_id during initialization

Workaround that Works

Step 1: Initial Run (will fail, but uploads snippet)

# Run Terraform - this will fail to provision the VM correctly
# but successfully uploads the cloud-init snippet
terraform init
terraform apply
# Note the VMID of the created VM (e.g., 100)

Step 2: Clean Terraform State

# Remove Terraform state files to prevent it from managing the first VM
rm -f terraform.tfstate*
rm -f .terraform.lock.hcl
rm -rf .terraform/

# The cloud-init snippet remains on the Proxmox node at:
# /var/lib/vz/snippets/user-data-cloud-config.yaml
# DO NOT DESTROY THE VM BEFORE SAVE THE SNIPPET

Step 3: Preserve snippet in the Proxmox host

# Create a copy of the snippet to use it in next iterations, e.g.
cd /var/lib/vz/snippets
cp user-data-cloud-config.yaml user-data-cloud-config.yaml.copy

Step 4: Second Run (will succeed)

# This will create a new VM with a different VMID (e.g., 101 instead of 100)

terraform init
terraform apply

# This time the VM will provision successfully on LVM storage

Critical Observations

What DOESN'T work:

• Using depends_on between proxmox_virtual_environment_file and proxmox_virtual_environment_vm
• Adding null_resource with delays/sleeps between resources
• Pre-creating an empty/dummy snippet file manually on the Proxmox node
• Destroying and recreating a VM with the same VMID (even with snippet present)
• Different lifecycle settings on the file resource

What DOES work:

• Snippet created by Terraform's first run (manual creation doesn't work)
• Creating a NEW VM with a different VMID in subsequent runs
• The snippet persists and is properly indexed after the first failed run

Key Finding: VMID Dependency

The VMID appears to be part of Proxmox's caching/indexing mechanism for cloud-init snippets.

Example scenario that FAILS:

1. First run: VM created with VMID 100 (fails, snippet uploaded)
2. Clean Terraform state
3. Preserve the snippet with a copy
4. Delete VM 100 from Proxmox manually
5. Restore the snippet from the copy
6. Second run: VM recreated with name that gets VMID 100 again → FAILS

Example scenario that SUCCEEDS:

1. First run: VM created with VMID 100 (fails, snippet uploaded)
2. Clean Terraform
3. Preserve the snippet with a copy
4. Preserve VM 100 in Proxmox. if you don´t delete the VM the original snippet remains too.
5. Second run: VM created with different VMID 101 → SUCCEEDS

[bpg-dev]

After extensive testing, I've identified the actual problem:

The snippet file is created successfully by the provider, but Proxmox doesn't properly register/index it for use in VM creation within the same Terraform apply cycle.

This seems pretty AI-sh to me, and I'm not convinced this is the only problem, or even the root cause. All the tests and examples I have here create the cloud-init snippet and the VM in one shot during the same apply.

The proxmox_virtual_environment_file resource successfully uploads the snippet to the datastore, but there's a synchronization issue between:

• The snippet being uploaded
• Proxmox indexing/registering it in the storage
• The VM being able to reference it via user_data_file_id during initialization

There's no such thing as "indexing" or "registering" a snippet, at least not that I'm aware of.

But again, why only 24.04 and LVM storage, if this is a snippet upload issue? Why 22.04 is not affected in the exact same scenario?

[rodfranck]

I honestly have no way of explaining why the problem only occurred with Ubuntu 24. The exact same script was successfully tested with cloud images of Ubuntu 22, Oracle Linux 9, and Debian 12, and only failed with Ubuntu 24 cloud images (several of them).

I found a very similar behavior described in this forum thread:
https://forum.proxmox.com/threads/template-on-shared-storage.146756

For context, our shared storage is a Dell PowerVault ME4024.

As a workaround, I decided to create the VMs on local-lvm first, and then migrate the disks to the SAN storage afterward. Although this required an additional manual step, it seemed acceptable if the end result matched the expected architecture.

However, after migrating the disks, they became unusable and the VM was effectively lost.

At this point, I asked my manager, who has strong expertise in infrastructure, networking, and security, to help investigate the issue. After analyzing the behavior together, we concluded that the cause might be related to the SAN itself.

It seemed as if something was being cached or retained on the storage side. When new VMs were created using the same VM IDs (regardless of how many times they were deleted and recreated during testing — successful or failed), the SAN appeared to reuse previously existing data instead of writing clean data consistently.

The VM ID proved to be a key detail, as demonstrated in the workaround of maintaining a previous VM to generate the new VM with a new ID.

During troubleshooting, we verified that each disk created on the SAN corresponds to a logical volume, and that the logical volume name includes the VM ID. This led us to review the LVM storage configuration in Proxmox.

We enabled the option:

“Wipe Removed Volumes”

This ensures that recently deleted logical volumes are properly wiped when removed.

After enabling this setting — VOILA — everything worked.

Disk migrations to the SAN started working correctly, and more important we were also able to create Ubuntu 24 VMs directly on the SAN storage without any issues, even using VM IDs that had never been used before.

Thank for your help @bpg-dev!

[bpg-dev]

That's quite a trip. Thanks for sharing!