QEMU MicroVM: why such bad results and why not in the Firecracker paper?

[dimakuv]

Dear Firecracker paper authors,

While looking at some other paper (rellermeyer/container_benchmarks#1), I noticed the unexplainably bad results of QEMU's MicroVM in your GitHub repo.

Here is the QEMU MicroVM command line: https://github.com/firecracker-microvm/nsdi2020-data/blob/cf1b30f674523ee44363f74e16bcb301ffb10598/scripts/util_start_qemu_uvm.sh

Here is the commit that added QEMU MicroVM data: b35147f

You also updated all the plots, some of them include QEMU MicroVM perf, especially this one: https://github.com/firecracker-microvm/nsdi2020-data/blob/master/data/m5d.metal/boot-serial-all.pdf

If I'm reading the plot correctly, and if I'm not confusing two shades of "light blue", then QEMU MicroVM exhibits extremely bad boot time (~600ms, about 2.5x worse than normal QEMU).

I also notice that your final paper https://www.usenix.org/system/files/nsdi20-paper-agache.pdf doesn't mention QEMU MicroVM at all (this may be just timing issues, since your GitHub updates are dated with 9. February and the NSDI conference was on 25. February).

So I have two questions:

• Do you have any explanations on QEMU MicroVM boot times? Did you maybe expand on this in some follow-up paper, video, blog, etc?
• How can you explain the fact that QEMU's MicroVM behaves worse than normal QEMU ( with normal machine type)?
• Why didn't you include any mentions of QEMU's MicroVM in the final paper?

[raduweiss]

Hi Dmitrii. Let us take a look here, and we'll get back to you in a few days. Thanks, Radu