monitor: One-to-one relationship between guests and images

[delan]

so far, we’ve managed the lifecycle of an image like servo-ubuntu2204 like this:

• rebuild the image by building a new profile guest (libvirt guest named “servo-ubuntu2204”)
  • destroy the guest with that name, if it exists
  • recreate the guest from profiles/servo-ubuntu2204/guest.xml
  • set the disk to /path/to/base/servo-ubuntu2204/base.img@[timestamp]
  • start the guest, so that it can install tooling, bake in cached builds, etc
  • destroy the libvirt guest
  • recreate the guest from profiles/servo-ubuntu2204/guest.xml
  • set the disk to /path/to/base/servo-ubuntu2204/base.img (symlink)
• if successful, symlink /path/to/base/servo-ubuntu2204/base.img to base.img@[timestamp]
• if unsuccessful, symlink /path/to/base/servo-ubuntu2204/base.img to base.img@[oldTimestamp]
• create runners by cloning that guest into each runner guest (“ci-runner-servo-ubuntu2204.[number]”)
  • clone the disks, but not the guests, with the monitor (so it can be parallelised)
  • clone the guest, but not the disks, with virt-clone

that works well enough for our current setup with libvirt and ZFS, but some hypervisors, like UTM (#64), will require us to treat each virtual machine as a self-contained object, with its own configuration and image files. in UTM, it’s hard to destroy a guest without also destroying its disks.

this patch moves the monitor closer to that world, by avoiding the destroy-recreate-configure pattern in image rebuilds. the lifecycle for servo-ubuntu2204 images now looks like this:

• rebuild the image by building a new rebuild guest (“ci-rebuild-servo-ubuntu2204@[timestamp]”)
  • create the guest from profiles/servo-ubuntu2204/guest.xml
  • set the disk to /path/to/base/servo-ubuntu2204/base.img@[timestamp]
  • start the guest, so that it can install tooling, bake in cached builds, etc
• if successful, convert that guest to a new template guest (“ci-template-servo-ubuntu2204@[timestamp]”)
  • conversion just means renaming the guest from “ci-rebuild-…” to “ci-template-…”
• if unsuccessful, just destroy that guest
• create runners by cloning that guest into each runner guest (“ci-runner-servo-ubuntu2204.[number]”)
  • clone the disks, but not the guests, with the monitor (so it can be parallelised)
  • clone the guest, but not the disks, with virt-clone

bonus: with this patch, failed rebuilds will no longer leave the template in an inconsistent state, with the old image data (good) but the new guest configuration (bad because it may be broken).

[delan]

deploying only ci2 and ci3 initially, since this is a complex change:

$ ( for i in 2 3; do ./do deploy ci$i & done )

[delan]

those look good, so deploying the rest now:

$ ( for i in 0 1 4; do ./do deploy ci$i & done )