QEMU execution engine #63
Description
It could potentially be useful to have a QEMU-based execution engine, for users that want something lighter weight than Docker on non-Linux machines. We used to have a QEMU backend in BinaryBuilder, which may be a good place to start. There were a few added complexities to this setup, that will need to be mapped over to the Sandbox.jl way of doing things:
-
We originally needed to use Keno's QEMU fork to get better 9p performance for read/write mounts. I assume that since it's been a few years, these commits have made it into a QEMU release and we can probably just use mainline QEMU, but it's worth checking.
-
We needed to create an
initreplacement to do things like set up mounts, define environment variables, etc... inside of the QEMU environment. We did this by adding an "initmode" to oursandbox.chelper. Combining these two very different execution types (user namespaces and qemu init) allowed us to share some code, but it needlessly complicated the tool, and I'd much rather have separatesandboxandqemu_initbinaries. If we need to, we can share some utility.cfiles between the two projects. Theinitmode did things like mount/dev, setup VM networking, and cleanup all state before powering off the emulator. -
Although QEMU helpfully maps
stdinandstdoutfor us, there's not an easy way for us to get command-line arguments or environment variables into the inside of the emulator, so we create our own communications channel by telling QEMU to create a fake serial device on the inside, and bind it to a unix domain socket on the outside, then serialize our metadata and transmit it to theinitreplacement on the inside. Theinitreplacement also needs to receive that information and deal with it accordingly.
It's entirely possible that there are easier ways to deal with this complexity now than we had back then, but this is how we used to do this. The number of features that Sandbox.jl supports are even greater than what BB used to support, and some of them are going to require some thought. The feature list is basically the list of things you can configure in a SandboxConfig. The main things to figure out are:
read_only_maps: In the past, we provided.squashfscompiler shards precisely because QEMU supported them. Since other executors such as Docker don't support those, we may need to come up with a solution that works well for both. It's possible that the9pfilesharing has become much faster now than it was before, testing will be needed.persist: how to best persist changes from one QEMU invocation to another. One possibility is to have a singleQEMUinstance running as long as theExecutorobject lives, then launch processes inside it by communicating over the serial socket. Another possibility is to have a host-backed filestore for all modifications made during execution, then map that in for subsequent executions.multiarch: This could be as simple as mapping to different QEMU host binaries, although we will of course need to provide differentLinuxandinitbundles for each target architecutre.