My ongoing migration to v0.9.0

[solemnwarning]

Hi all,

I have a somewhat large and complex libvirt deployment of Linux, Windows and macOS guests across multiple VM hosts (kvm-build-environment) and have started working on migrating it to use terraform-provider-libvirt, these are my observations from migrating a small number of Linux VMs.

I went through this by taking an existing deployment, upgrading the plugin (with terraform init -upgrade) and then tweaking the HCL files where any errors highlighted in terraform plan based on the examples in the documentation (which perhaps needs some corrections/more examples, or maybe I just didn't read it properly).

This list is hopefully helpful to other users, and as feedback to the developer - don't take this as complaints, thanks for all the work you've put into this!

Issues I ran into

libvirt_domain resource migration

The existing 0.8 libvirt_domain resources caused an error when running terraform plan, in order to work around this I had to delete the domain resources from the terraform state (with terraform state rm) and from the hypervisor (via virt-manager).

I don't have the error handy since it already scrolled off the top of my terminal, sorry.

Attaching a qcow2 image to a domain

Based on the documentation, I wound up with lightly modified libvirt_volume definitions for the main system disk:

resource "libvirt_volume" "root" {
  name   = "${ var.hostname }.${ var.domain }_root.qcow2"
  pool   = var.storage_pool
  format = "qcow2"

  create = {
    content = {
      url = "${ path.root }/vcpkg-cache-server-image/builds/${ local.image_version }/vcpkg-cache-server.qcow2"
    }
  }
}

And this device definition to expose it to the VM:

  devices = {
    disks = [
      {
        device = "disk"
        target = { dev = "vda" }
        source = { pool = libvirt_volume.root.pool, volume = libvirt_volume.root.name }
      }
    ]

My first attempt defined the source block with the file directive as per the libvirt_cloudinit_disk example, which resulted in the qcow2 image being mounted as a raw image, which obviously didn't work.

Attaching a cloud-init disk to the domain

First observation: the meta_data parameter is now mandatory, but I don't use that, so I just defined it as an empty string, is there a technical reason this is mandatory now, or would it make sense to return to the 0.8.x behaviour of it being optional?

The creation of the ISO and uploading it to the hypervisor has been split into two blocks, so I removed the pool directive from the libvirt_cloudinit_disk block and added a libvirt_volume block to upload the resultant ISO:

resource "libvirt_volume" "cloudinit" {
  name = "${ var.hostname }.${ var.domain }_cloud-init.iso"
  pool = var.storage_pool

  create = {
    content = {
      url = libvirt_cloudinit_disk.cloudinit.path
    }
  }
}

The documentation suggests defining format = "raw" in the libvirt_volume block, however this results in internal error warnings from terraform due to the format implicitly getting changed to "iso" when applying, and unnecessary domain changes every time apply is run.

And a device definition to attach the ISO to the VM:

  devices = {
    disks = [
      {
        device = "cdrom"
        target = { dev = "hda" }
        source = { file = libvirt_volume.cloudinit.path }
      }
    ]

Graphical console support

To get a graphical console as per default in 0.8.x, I added the following devices to the VM block, which was a process of guesswork and trial-and-error since there appears to be no example in the documentation:

  devices = {
    video = {
      type = "cirrus"
    }

    graphics = {
      spice = {
        autoport = "yes"
      }
    }
  }

There is one minor difference to the previous behaviour here - the listen address of the Spice server winds up set to 127.0.0.1 rather than unset (i.e. "hypervisor default"), which works fine for me, but perhaps might be a problem for some deployments. The listen directive takes a string rather than a set of parameters like the libvirt XML (which doesn't seem to get mapped correctly), so I don't think the previous can be specified right now.

Memory is specified in KiB by default now.

Change memory = 2048 to memory = 2048, unit = "MiB" otherwise your domain will suddenly find itself with 2MiB of RAM rather than 2GiB.

No way to specify qemu guest agent channel

This has already been discussed (#1231) and it looks like work is underway.

No way to specify input devices

Despite many other things no longer being provided by default, VMs come with PS/2 keyboards and mice attached, as far as I can tell, there is no way to override this from HCL, which will make macOS not feasible (only supports USB), and graphical guests annoying (relative mouse-based input rather than absolute tablet-based input).

Other minor notes

The domain type now needs to be explicitly set in the libvirt_domain block:

  os = {
    type = "hvm"
  }

The random device was previously added by default but must now be specified in the devices block:

devices = [
    rngs = [
      {
        device = "/dev/urandom"
        model = "virtio"
      }
    ]
  }

Both of the RNG parameters (device and model) are marked as optional and defaulting to the values shown above, however omitting either results in another internal error warning from terraform and unnecessary changes when running terraform apply.

The network interfaces are now specified within the devices block, in a slightly more verbose form:

devices = [
    interfaces = [
      {
        type = "bridge"
        source = { bridge = "dmz-build" }
        model = "virtio"
      },
    ]
  ]

Changing running to false now shuts down running domains

This is a behaviour change from 0.8.x, and not desirable in some of my use cases - can we get an option to make terraform-provider-libvirt create domains without starting but leave existing running ones alone?