How to get qemu guest agent and text based console to work with latest 0.9.0 release?

[mrmcmuffinz]

Below is the code I have so far for standing up my virtual machines.

#####################################
# Locals
#####################################
locals {
  image_filename   = "ubuntu-${var.ubuntu_release}-server-cloudimg-${var.image_arch}.img"
  hostnet_cidr     = var.hostnet_subnet
  cloud_init_dir   = "${path.module}/cloud-init"
  base_url         = "https://cloud-images.ubuntu.com/releases/${var.ubuntu_series}/${var.ubuntu_release_build}"
  image_url        = "${local.base_url}/${local.image_filename}"
}

#####################################
# Host-only Network (virbr1)
#####################################
resource "libvirt_network" "hostnet" {
  name      = "hostnet"
  mode      = "nat"
  #autostart = true

  ips = [
    {
      address = "192.168.50.1"
      prefix  = 24
      dhcp = {
        enabled = true
        ranges = [
          {
            start = "192.168.50.2"
            end = "192.168.50.254"
          }
        ]
      }
    }
  ]
}

#####################################
# Per-node Disks based on the Base Image
#####################################
resource "libvirt_volume" "node_disks" {
  for_each = var.nodes

  name     = "${each.key}.qcow2"
  pool     = "default"
  format   = "qcow2"
  create   = {
    content = {
      url = local.image_url
    }
  }
}

#####################################
# Cloud-Init (user-data + network-config) per node
#####################################
resource "libvirt_cloudinit_disk" "node_ci" {
  for_each = var.nodes
  name     = "${each.key}-seed.iso"

  meta_data = templatefile("${local.cloud_init_dir}/meta.yaml.tmpl", {
    instance_id = each.key
    hostname    = each.key
  })

  user_data = templatefile("${local.cloud_init_dir}/node.yaml", {
    ssh_pubkey = file(var.ssh_pubkey)
    node_name  = each.key
    node_role  = each.value.role
    node_ip    = each.value.ip_address
  })

  network_config = templatefile("${local.cloud_init_dir}/net.yaml.tmpl", {
    ip_address = each.value.ip_address
    mac_lan     = lower(each.value.mac)          # br0 MAC
    mac_hostnet = lower(each.value.hostnet_mac)  # hostnet MAC
  })
}

#####################################
# Stable Cloud-init ISO copied into default pool
# (avoids /tmp ISO + source.volume drift bugs)
#####################################
resource "libvirt_volume" "seed_iso" {
  for_each = var.nodes

  name   = "${each.key}-seed.iso"
  pool   = "default"
  format = "iso"

  create = {
    content = {
      url = libvirt_cloudinit_disk.node_ci[each.key].path
    }
  }

  depends_on = [libvirt_cloudinit_disk.node_ci]
}

#####################################
# Domains (VMs)
#####################################
resource "libvirt_domain" "nodes" {
  for_each = var.nodes

  name   = "txgrid-${each.key}"
  type   = "kvm"
  memory = each.value.memory
  unit   = "MiB"
  vcpu   = each.value.vcpu

  running = true
  autostart = true

  os = {
    type    = "hvm"
    arch    = "x86_64"
    machine = "q35"
  }

  devices = {
    disks = [
      {
        device = "disk"
        source = {
          pool = libvirt_volume.node_disks[each.key].pool
          volume = libvirt_volume.node_disks[each.key].name
        }
        target = {
          dev = "vda"
          bus = "virtio"
        }
      },
      {
        device   = "cdrom"
        readonly = true
        source = {
          pool = libvirt_volume.seed_iso[each.key].pool
          volume = libvirt_volume.seed_iso[each.key].name
        }
        target = {
          dev = "sda"
          bus = "sata"
        }
      }
    ]

    interfaces = [
      {
        type   = "bridge"
        model  = "virtio"
        mac    = lower(each.value.mac)
        source = { bridge = "br0" }
      },
      {
        type   = "network"
        model  = "virtio"
        mac    = lower(each.value.hostnet_mac)
        source = { network = libvirt_network.hostnet.name }
      }
    ]

    graphics = {
      spice = {
        autoport = "yes"
        listen   = "127.0.0.1"
      }
    }
    video = {
      type = "virtio"
    }
  }
}

resource "time_sleep" "wait_for_nodes_600s" {
  depends_on = [libvirt_domain.nodes]
  create_duration = "600s"
}

locals {
  node_networks = [
    for n in libvirt_domain.nodes : n
  ]
}

I'm able to login to the virtual machines using Virtual Machine Manager however I can't seem to do it over console. I also have to go a round about way of logging in via gui to find out the network information and then I can SSH to my KVM. The issue is I used to be able to run the below command but since the channel is not configured to allow the KVM to communicate with the agent it fails.

$ virsh domifaddr --source agent txgrid-cp0
error: Failed to query for interfaces addresses
error: argument unsupported: QEMU guest agent is not configured

I have tried looking for examples and read through the docs. I have had no luck and it isn't clear to me how to proceed.

In summary

1. How do I fix console access?
2. How can I fix the qemu guest agent not being able to connect?

[mrmcmuffinz]

Just as an fyi, I do have working code https://github.com/mrmcmuffinz/my-ai-journey/blob/v0.1.1/infra/terraform/main.tf but that uses provider 0.8.3 which I have read in numerous issues it has been replaced with 0.9.0.

[bamhm182]

I'm also trying to figure out these issues while converting from 0.8.3 to 0.9.0. I THINK this will solve your console issue. Still trying to figure out the agent issue.

resource "libvirt_domain" "nodes" {
  ...
  devices = [
    ...
    consoles = [
      {
        type = "pty"
        target = {
          type = "serial"
          port = 0
        }
      }
    ]
  ]
}

[bamhm182]

This is the XML that's required in order to connect the agent:

<devices>
  <channel type='unix'>
    <source mode='bind'/>
    <target type='virtio' name='org.qemu.guest_agent.0'/>
  </channel>
</devices>

I determined this by playing around with Virtual Machine Manager, and confirmed it by having v0.9.0 create a VM, modifying the XML, giving it a reboot, and confirming the agent is working. That means that the following SHOULD work:

devices = {
  ...
  channels = [
    {
      type = "unix"
      source = {
        mode = "bind"
      }
      target = {
        type = "virtio"
        name = "org.qemu.guest_agent.0"
      }
    }
  ]
}

However, that isn't working for me. I don't see anything that references any of the devices (except interfaces) by name, so it isn't a function of the provider that is including/excluding it. I think there's some code around cleaning up the devices to remove invalid things, maybe it's hitting on that? I need to dig into it further...

[bamhm182]

Smooth brain moment. The reason I couldn't find anything killing it, yet it wasn't being processed, is because I was looking at the wrong version of the code. In v0.9.0, only defined device types (seen here) can be defined. Channels is not in that list. Unless I'm missing something, I don't see a way that we could define Channels in v0.9.0. The good news (if I'm understanding it correctly) is that the next release (and potentially one you could build from main) is going to be a lot more flexible and allow us to do whatever we want, then just check if it looks close enough to being valid.

[mrmcmuffinz]

I have attempted all the prior suggestions with release 0.9.0 but it does not work. In fact one weird behavior I noticed is that you can actually define channels as a device but it does absolutely nothing. Which I think is a bug.

relevant domain code:

#####################################
# Domains (VMs)
#####################################
resource "libvirt_domain" "nodes" {
  for_each = var.nodes

  name   = "txgrid-${each.key}"
  type   = "kvm"
  memory = each.value.memory
  unit   = "MiB"
  vcpu   = each.value.vcpu

  running = true
  autostart = true

  os = {
    type    = "hvm"
    arch    = "x86_64"
    machine = "q35"
  }

  devices = {
    disks = [
      {
        device = "disk"
        source = {
          pool = libvirt_volume.node_disks[each.key].pool
          volume = libvirt_volume.node_disks[each.key].name
        }
        target = {
          dev = "vda"
          bus = "virtio"
        }
      },
      {
        device   = "cdrom"
        readonly = true
        source = {
          pool = libvirt_volume.seed_iso[each.key].pool
          volume = libvirt_volume.seed_iso[each.key].name
        }
        target = {
          dev = "sda"
          bus = "sata"
        }
      }
    ]

    interfaces = [
      {
        type   = "bridge"
        model  = "virtio"
        mac    = lower(each.value.mac)
        source = { bridge = "br0" }
      },
      {
        type   = "network"
        model  = "virtio"
        mac    = lower(each.value.hostnet_mac)
        source = { network = libvirt_network.hostnet.name }
      }
    ]

    channels = [
      {
        type = "unix"
        target_type = "virtio"
        target_name = "org.qemu.guest_agent.0"
      }
    ]

    #    serials = [
    #      {
    #        type = "pty"
    #        target_port = "0"
    #      }
    #    ]
    #
    #    consoles = [
    #      {
    #        type = "pty"
    #        target_type = "serial"
    #        target_port = "0"
    #      }
    #    ]

    graphics = {
      spice = {
        autoport = "yes"
        listen   = "127.0.0.1"
      }
    }
    video = {
      type = "virtio"
    }
  }
}

virsh dumpxml txgrid-cp0

<domain type='kvm' id='83'>
  <name>txgrid-cp0</name>
  <uuid>2f3e6291-673f-48cc-8024-60867f8d1bcd</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source pool='default' volume='cp0.qcow2' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='volume' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source pool='default' volume='cp0-seed.iso' index='1'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <interface type='bridge'>
      <mac address='34:97:f6:aa:bb:c0'/>
      <source bridge='br0'/>
      <target dev='vnet164'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:b2:ce:4f'/>
      <source network='hostnet' portid='1e63ab08-3b6d-40eb-8b87-d5d943b48e8e' bridge='virbr0'/>
      <target dev='vnet165'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='spice' port='5901' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <audio id='1' type='spice'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-2f3e6291-673f-48cc-8024-60867f8d1bcd</label>
    <imagelabel>libvirt-2f3e6291-673f-48cc-8024-60867f8d1bcd</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+993</label>
    <imagelabel>+64055:+993</imagelabel>
  </seclabel>
</domain>

[hiousi]

this one worked for me, I'm using main branch:

channels = [
      {
        source = {
          unix = {}
        }
        target = {
          virt_io = {
            name  = "org.qemu.guest_agent.0"
          }
        }
      }
    ]

[mrmcmuffinz]

How did you switch over to main branch? I tried switching by

1. install goenv and goenv global 1.25.4
2. clone repo libvirt terraform provider.
3. run make install
4. go back to my code directory
5. clear out terraform lock files.
6. rm -rf .terraform folder
7. terraform init

However when I do that my terraform code doesn't even get past the plan phase. Atleast with 0.9.0 I could actually provision something but with latest main code it breaks.

My code is at https://github.com/mrmcmuffinz/my-ai-journey/blob/upgrade_libvirt_provider_to_0_9_x/infra/terraform/main.tf

[hiousi]

I hit the same issue when moving to the main branch. What helped was extracting the provider schema with:

terraform providers schema -json

Then I used jq to compare it with the old version. A few attributes/blocks changed, so Terraform failed at plan until I updated my config to match the new schema. Reviewing the schema diff should show what’s breaking your setup.