applehv can silently downgrade the amount of memory

[Salynn]

The investigation in containers/podman#25803 has uncovered that this issue can be reproduced with vfkit itself, so it's not an issue of how podman is using vfkit.

Under some circumstances, applehv backed VMs created by vfkit can silently have the amount of RAM seen by the OS downgraded.

Here is what's known:

• This has not been reproduced for any memory sizes below 63GiB + 1MiB
• This has not been reproduced on a machine with less than 64GiB of RAM
• This has (so far) only been reproduced on Intel CPUs (but it's unclear if the attempt on an ARM CPU had sufficient RAM)
• Virtualization Framework correctly rejects with an error a request for more memory than the host has
• A slight modification to the Running Linux in a Virtual Machine sample code could not reproduce the issue and was able to create a VM that exposed 64GiB to the guest OS

So far, this issue has only been reproduced with coreos (as used by podman) and Fedora Cloud 42.

I have not yet been able to get vfkit to launch any other image to a point where I could access the system to check on how much memory the OS saw, so I have not been able to test other distributions. Recommendations for other disk images to try would be appreciated.

[cfergeau]

This has (so far) only been reproduced on Intel CPUs (but it's unclear if the attempt on an ARM CPU had sufficient RAM)

I tested this on an m1 with 64GiB but could not reproduce, see containers/podman#25803 (comment)

[cfergeau]

The steps to reproduce are:

• download a fedora cloud image from https://fedoraproject.org/cloud/download?beta. Pick the arch corresponding to your mac, and use a "Raw" image
• start it with https://github.com/crc-org/vfkit/blob/main/contrib/scripts/start-with-cloud-init.sh
• ssh into the VM and run free, check the total column

I have not yet been able to get vfkit to launch any other image to a point where I could access the system to check on how much memory the OS saw, so I have not been able to test other distributions. Recommendations for other disk images to try would be appreciated.

fedora coreos images should work with https://github.com/crc-org/vfkit/blob/main/contrib/scripts/start-with-ignition.sh , if they don’t, @vyasgun should be able to help you.

I’m not sure these have been tested with start-with-cloud-init.sh, but these ubuntu images should support cloud-init https://cloud-images.ubuntu.com/

And there’s also puipui linux which is totally different. Unpack the archive, start vfkit with --cpus 2 --memory 64512 --bootloader linux,kernel=bzImage,initrd=initramfs.cpio.gz,cmdline="console=hvc0" --device virtio-net,nat,mac=52:54:00:70:2b:71
Then the IP of the VM will be in /var/db/dhcpd_leases and you can ssh using root/passwd
Not sure free will be available on the VM though :-/

[gbraad]

Dynamic allocation of memory is not uncommon for hypervisors (eg. Hyper-V), but is this an issue? It is more important why it does so (memory pressure on the host?) and if this will also scale up again when needed/possible.

[cfergeau]

Recommendations for other disk images to try would be appreciated.

You could also give a try to lima with the vz driver https://lima-vm.io/docs/config/vmtype/#vz
It uses the same go bindings as vfkit (Code-Hex/vz), but the code on top of it is different from vfkit.

[Salynn]

fedora coreos images should work with https://github.com/crc-org/vfkit/blob/main/contrib/scripts/start-with-ignition.sh , if they don’t, @vyasgun should be able to help you.

I tried the coreos with ignition and could not gain access. Not sure if it was an issue with the coreos image or with the parameters I was passing to the ignition file or something else I was doing wrong. I had the same issue with cloud-init with Fedora Cloud 41. Fortunately, Fedora Cloud 42 worked, so 🤷🏻.

I’m not sure these have been tested with start-with-cloud-init.sh, but these ubuntu images should support cloud-init https://cloud-images.ubuntu.com/

I just tried a sampling and could not find an image that would boot. There's probably some parameters to vfkit that could be changed to make one of them work, but I'm not sure what.

And there’s also puipui linux which is totally different. Unpack the archive, start vfkit with --cpus 2 --memory 64512 --bootloader linux,kernel=bzImage,initrd=initramfs.cpio.gz,cmdline="console=hvc0" --device virtio-net,nat,mac=52:54:00:70:2b:71 Then the IP of the VM will be in /var/db/dhcpd_leases and you can ssh using root/passwd Not sure free will be available on the VM though :-/

That worked! And did not exhibit the issue. But neither did my usage of the Fedora kernel and initrd directly (as described in the Running Linux in a Virtual Machine sample code documentation).

That might possibly mean this could be a Virtualization Framework issue related to how the boot image is specified? Or it could mean it has something to do with a more full Fedora system than just the initrd environment?

P.S. free is not necessary to validate the issue. MemTotal in /proc/meminfo can also be used to check.

[Salynn]

Recommendations for other disk images to try would be appreciated.

You could also give a try to lima with the vz driver https://lima-vm.io/docs/config/vmtype/#vz It uses the same go bindings as vfkit (Code-Hex/vz), but the code on top of it is different from vfkit.

Excellent suggestion. Using limactl start --cpus 2 --memory 64 --vm-type vz template://fedora I was able to reproduce the issue. Same with limactl start --cpus 2 --memory 64 --vm-type vz template://ubuntu-lts (also reproduced the issue). So that seems to rule out the OS as the likely culprit, and we're back to a bad behavior in Virtualization Framework, possibly triggered by the type of bootloader provided?

(The ubuntu template failed to boot with functioning ssh.)

[cfergeau]

So that seems to rule out the OS as the likely culprit, and we're back to a bad behavior in Virtualization Framework, possibly triggered by the type of bootloader provided?

Could indeed be a bug in Apple’s UEFI implementation. I’ll look how we can report it to them.

[cfergeau]

If you want to keep poking at this, one more test you can do is to start a VM where you can see the bug, copy its kernel/initrd to the host (they should be in /boot), write down its kernel commandline (cat /proc/cmdline), and then try to start it the same way as you started the puipui linux VM, adjusting the paths/cmdline to use the ones you just copied. You’ll need to add --device virtio-blk,path=/Users/virtuser/vfkit.img or so to the cmdline to specify the disk image.

[Salynn]

If you want to keep poking at this, one more test you can do is to start a VM where you can see the bug, copy its kernel/initrd to the host (they should be in /boot), write down its kernel commandline (cat /proc/cmdline), and then try to start it the same way as you started the puipui linux VM, adjusting the paths/cmdline to use the ones you just copied. You’ll need to add --device virtio-blk,path=/Users/virtuser/vfkit.img or so to the cmdline to specify the disk image.

@cfergeau you elaborate on what you mean by

--device virtio-blk,path=/Users/virtuser/vfkit.img or so

I don't seem to have a generic vfkit.img, and I'm not sure what disk image to use with the extracted kernel and initramfs. Using the raw disk image from which I extracted them doesn't work.

In the meantime, I have used the aforementioned sample program from Apple to launch a VM with the extracted kernel and initramfs. Because it had no disk, it booted to a dracut recovery shell. From there I was able to confirm the OS did see the correct amount of memory. I'm not sure if that qualifies as a sufficient test case here or not, though.

[cfergeau]

I don't seem to have a generic vfkit.img, and I'm not sure what disk image to use with the extracted kernel and initramfs. Using the raw disk image from which I extracted them doesn't work.

I was referring to the raw disk image you extracted from, not sure why it did not work...

Because it had no disk, it booted to a dracut recovery shell.

You should be able to get that recovery shell either using --device virtio-input,keyboard --device virtio-input,pointing --device virtio-gpu,width=1920,height=1080 --gui (https://github.com/crc-org/vfkit/blob/main/doc/usage.md#enabling-a-graphical-user-interface) or --device virtio-serial,stdio

But at this point, it’s imo less and less important to spend time on this, especially as this likely takes more and more time :-/

[cfergeau]

In the meantime, I have used the aforementioned sample program from Apple to launch a VM with the extracted kernel and initramfs. Because it had no disk, it booted to a dracut recovery shell. From there I was able to confirm the OS did see the correct amount of memory. I'm not sure if that qualifies as a sufficient test case here or not, though.

Actually you should be able to get the same dracut recovery shell by adding rd.break=initqueue to the kernel command line arguments and using --device virtio-input,keyboard --device virtio-input,pointing --device virtio-gpu,width=1920,height=1080 --gui to get a graphical console.

[Salynn]

Actually you should be able to get the same dracut recovery shell by adding rd.break=initqueue to the kernel command line arguments and using --device virtio-input,keyboard --device virtio-input,pointing --device virtio-gpu,width=1920,height=1080 --gui to get a graphical console.

@cfergeau To make sure I understand, you're suggesting to check the recovery shell in the problematic VM and confirm it indicates the issue still?