Skip to content

Conversation

@ArrayBolt3
Copy link
Contributor

@ArrayBolt3 ArrayBolt3 commented Apr 5, 2025

Adds Raspberry Pi installation and image generation support. Cross-builds are supported (you can create a working Raspberry Pi image on an amd64 machine).

Some notes:

  • raspi-firmware generates the cmdline.txt for the kernel in a highly non-ideal way, using a device path rather than a UUID for identifying the root directory. This has to be manually patched to get the image to boot at all.
  • If your host system is a Pi using one of the raspi.debian.net images, you probably want to pass --defaultinterfaces to grml-debootstrap, since the /etc/network/interfaces file used in the raspi.debian.net images is just a pointer to /etc/network/interfaces.d, which grml-debootstrap doesn't copy over. Using --defaultinterfaces will prevent this from being a problem, not using it will mean you'll have no network when you boot the system.
  • The image DOES NOT automatically resize itself on first boot. The RPi image generation is more like VM image generation, the image is fixed-size and you have to resize it afterwards. If you don't want that, you can install directly to a USB drive or SD card.
  • This uses the current way of doing things as implemented in the normal Debian Raspberry Pi images, where the kernel is booted directly by the firmware bootloader. There are lots of other ways to boot a Pi (involving u-boot and/or edkII), which might be more suitable for some use cases. However, as grml-debootstrap is intended to install pure Debian, I tried to stick with a pure Debian approach as much as possible. I've also done research showing that a direct-kernel-boot install like this can be converted to a U-Boot + GRUB install without too much effort. (See https://www.kicksecure.com/wiki/Dev/boot#Booting_Debian_Trixie_with_GRUB_.2B_u-boot_on_Raspberry_Pi_4)

Fixes #114

@ArrayBolt3 ArrayBolt3 marked this pull request as draft April 5, 2025 00:24
@ArrayBolt3
Copy link
Contributor Author

Found a bug in grml-debootstrap that existed before this PR - installing a VM image to a device such as /dev/sda was impossible because grml-debootstrap was trying to format devices named /dev/mapper/sda1p1 (which is wrong for multiple reasons). This was because it always assumed that device names were formatted as {disk_driver}{disk_number}p{partition_number}, when some devices are formatted as {disk_driver}{disk_letter}{partition_number}. Fixed in an added commit (752a611), that one should be able to be cherry-picked without merging this entirely.

Copy link
Member

@zeha zeha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some initial comments, without any further consideration


# RPi images *require* non-free-firmware as the raspi-firmware package resides
# there, and Raspberry Pis are not bootable without it.
if [ -z "$_opt_non_free" ] && [ "$RPI" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this will need a warning, or better it should abort if non-free(-firmware) was not requested

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently there isn't an option for just enabling non-free-firmware but not the entirety of non-free. Should that be implemented, or is the job --non-free already does good enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an error message with bailout if --non-free isn't specified. If we want something that only enables non-free-firmware when possible, I'd be happy to add that too.

@zeha
Copy link
Member

zeha commented Apr 7, 2025

Found a bug in grml-debootstrap that existed before this PR - installing a VM image to a device such as /dev/sda was impossible because grml-debootstrap was trying to format devices named /dev/mapper/sda1p1 (which is wrong for multiple reasons). This was because it always assumed that device names were formatted as {disk_driver}{disk_number}p{partition_number}, when some devices are formatted as {disk_driver}{disk_letter}{partition_number}. Fixed in an added commit (752a611), that one should be able to be cherry-picked without merging this entirely.

AFAIK, VM images were never supposed to be written to devices by grml-debootstrap. @mika ?

@ArrayBolt3
Copy link
Contributor Author

ArrayBolt3 commented Apr 7, 2025

@zeha By installing a VM image directly to a device, I mean using a command line such as --target /dev/sda --vm, not using --vmfile. I probably just used the wrong terminology.

@mika
Copy link
Member

mika commented Apr 8, 2025

@zeha By installing a VM image directly to a device, I mean using a command line such as --target /dev/sda --vm, not using --vmfile. I probably just used the wrong terminology.

Hm, what's the use case for --target /dev/sda --vm exactly, as in: why install an VM into a block device at all and not into a plain file only? (Not claiming that there's no use case for it, but to clarify actual needs and how far we should go with supporting that :))

@ArrayBolt3
Copy link
Contributor Author

ArrayBolt3 commented Apr 8, 2025 via email

@mika
Copy link
Member

mika commented Apr 8, 2025

Off the top of my head, I know people oftentimes store VMs in LVM volumes rather than in files. I don't know for sure why, but I suspect performance plays a role. For the Raspberry Pi case specifically, being able to install directly to a physical device would be useful to prevent someone from having to generate an image, flash the image, and then resize the image to occupy the entire disk. They can just install straight to the device and use it.

Ah I see, thx4info!

@zeha
Copy link
Member

zeha commented Apr 8, 2025

JFTR, this is even documented in man grml-debootstrap:

  --vm
      Set up a Virtual Machine on an existing block device, which will be partitioned. This
      allows deployment of a Virtual Machine. The options needs to be combined with the
      --target option. This option automatically enables the --defaultinterfaces option. 
      Usage example: --vm --target /dev/mapper/your-vm-disk

(Note the example.)

@ArrayBolt3 ArrayBolt3 force-pushed the arraybolt3/rpi branch 2 times, most recently from dcfd735 to 39411c8 Compare April 9, 2025 22:33
@ArrayBolt3
Copy link
Contributor Author

Ran thorough tests to ensure things worked as expected. "Working" is defined as "the build doesn't error out, the produced image is bootable without having to take any special steps, the root user account can be logged into, the contents of /boot are appropriate for the installation being tested, the partition layout is appropriate for the installation being tested, and apt update successfully downloads an updated package database, ensuring network connectivity works. It may be required to modify /etc/network/interfaces to get the interface name right, this is expected."

  • Builder hardware: x86_64 VM
    • [x] Can create a working x86_64 BIOS VM image? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch amd64 --vmfile --target ./x86_64-bios-bookworm.img
    • [x] Can create a working x86_64 UEFI VM image? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch amd64 --vmfile --vmefi --target ./x86_64-efi-bookworm.img
    • [x] Can create a working x86_64 BIOS VM direct install? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch amd64 --vm --target /dev/vdb
    • [x] Can create a working x86_64 UEFI VM direct install? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch amd64 --vm --vmefi --target /dev/vdb
    • [x] Can create a working arm64 UEFI VM image? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch arm64 --vmfile --vmefi --target ./arm64-efi-bookworm.img
    • [x] Can create a working arm64 UEFI VM direct install? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch arm64 --vm --vmefi --target /dev/vdb
    • [x] Can create a working RPi image? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --rpifile --non-free --target ./arm64-rpi-bookworm.img
    • [x] Can create a working RPi direct install? sudo grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --rpi --non-free --target /dev/vdb
  • Builder hardware: arm64 Raspberry Pi 4
    • [x] Can create a working arm64 UEFI VM image? grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch arm64 --vmfile --vmefi --target ./arm64-efi-bookworm.img
    • [x] Can create a working arm64 UEFI VM direct install? grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --arch arm64 --vm --vmefi --target /dev/sda
    • [x] Can create a working RPi image? grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --rpifile --non-free --target ./arm64-rpi-bookworm.img
    • [x] Can create a working RPi direct install? grml-debootstrap --release bookworm --password 'x' --force --defaultinterfaces --rpi --non-free --target /dev/sda

I did fix a couple of bugs in the original PR as I went, but both fixes required small, non-invasive changes.

@ArrayBolt3 ArrayBolt3 marked this pull request as ready for review April 9, 2025 22:37
@ArrayBolt3
Copy link
Contributor Author

Kicksecure isn't really thrilled with the boot process that raspi-firmware uses by default, due to its effects on things like booting into older kernels, and customizing kernel parameters at boot time. If possible, we'd like to be using U-Boot and grub-efi-arm64 instead. I filed some feature requests upstream (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1102607, https://salsa.debian.org/raspi-team/image-specs/-/issues/78) to see if this can be changed. If things do change in that area and GRUB + U-Boot booting for the Pi is officially supported in Debian, could grml-debootstrap use that boot flow instead, or at least offer it as an option? (Obviously we're more than happy to send the patch for this, I just want to make sure this will be acceptable in the future so we can do project planning.)

@ArrayBolt3
Copy link
Contributor Author

@mika, @zeha Gentle ping, anything else that needs done to this?

@mika
Copy link
Member

mika commented Apr 18, 2025

@mika, @zeha Gentle ping, anything else that needs done to this?

I'm personally not really a huge fan of Raspberry devices, but I assume there are users out there who might benefit from it. So IMO it's also about how intrusive this change would become, and how we can ensure that things keep working.
Looking at e.g. https://github.com/ptrsr/pi-ci, is this something we could somehow also provide through our GitHub CI? 🤔

@ArrayBolt3
Copy link
Contributor Author

I'm not sure, I'm not exactly handy with Docker, and based on the documentation it looks like pi-ci is both an emulator and image generator, there doesn't appear to be a documented easy way to boot a custom image with it. Newer versions of QEMU have support for emulating the Raspberry Pi 4B, which might be able to be used somehow, but even then you have to specify the kernel and initramfs to boot directly (QEMU doesn't emulate the boot firmware's behavior of reading config.txt and figuring out what to boot from that). I guess it would be possible to kinda sorta emulate the boot firmware by mounting the build image, finding config.txt, parsing out the kernel and initramfs lines, copying those out of the image, then unmounting the image and booting the kernel + initramfs + image with QEMU. I'd be willing to give that a shot, though be warned that I'm not particularly experienced with CI systemd either :P

@adrelanos
Copy link
Contributor

It's published on dockerhub so the CI image does not have to be built from source code at least. Not sure that is still relevant...

Newer versions of QEMU have support for emulating the Raspberry Pi 4B, which might be able to be used somehow, but even then you have to specify the kernel and initramfs to boot directly (QEMU doesn't emulate the boot firmware's behavior of reading config.txt and figuring out what to boot from that).

So such as test would not directly test if either config.txt based kernel direct boot is functional. Let alone if GRUB based booting is functional.

I guess it would be possible to kinda sorta emulate the boot firmware by mounting the build image, finding config.txt, parsing out the kernel and initramfs lines, copying those out of the image, then unmounting the image and booting the kernel + initramfs + image with QEMU.

That sounds rather complex and may not be very useful as it would not test on the CI if the image is actually bootable. It would test kernel + initramfs and anything after that. Not sure how useful that would be?

Is CI testing mandatory?

@mika
Copy link
Member

mika commented Apr 23, 2025

That sounds rather complex and may not be very useful as it would not test on the CI if the image is actually bootable. It would test kernel + initramfs and anything after that. Not sure how useful that would be?

Is CI testing mandatory?

What we like and appreciate about CI testing is, that we catch any possible side-effects/breakages with any changes within g-d very early. It definitely improves our options to redesign/rewrite things, as well as accept contributions.
It's not a 100% "must have" for everything all the time, but it's something we definitely prefer having. :)

@zeha
Copy link
Member

zeha commented Apr 24, 2025

That sounds rather complex and may not be very useful as it would not test on the CI if the image is actually bootable. It would test kernel + initramfs and anything after that. Not sure how useful that would be?
Is CI testing mandatory?

What we like and appreciate about CI testing is, that we catch any possible side-effects/breakages with any changes within g-d very early. It definitely improves our options to redesign/rewrite things, as well as accept contributions. It's not a 100% "must have" for everything all the time, but it's something we definitely prefer having. :)

To add my thought: everything that's not CI tested will break. When we'll do a refactor, nobody will test stuff that's not automatically tested, and it will just be broken. My personal view is that it's close to mandatory.

@mika
Copy link
Member

mika commented Apr 24, 2025

That sounds rather complex and may not be very useful as it would not test on the CI if the image is actually bootable. It would test kernel + initramfs and anything after that. Not sure how useful that would be?
Is CI testing mandatory?

What we like and appreciate about CI testing is, that we catch any possible side-effects/breakages with any changes within g-d very early. It definitely improves our options to redesign/rewrite things, as well as accept contributions. It's not a 100% "must have" for everything all the time, but it's something we definitely prefer having. :)

To add my thought: everything that's not CI tested will break. When we'll do a refactor, nobody will test stuff that's not automatically tested, and it will just be broken. My personal view is that it's close to mandatory.

nod && ACK!

@ArrayBolt3
Copy link
Contributor Author

ArrayBolt3 commented Apr 24, 2025

kk, then I guess I'll be adding CI to this 👍

@ArrayBolt3
Copy link
Contributor Author

Unfortunately, it looks like CI is going to end up being very slow if we want to use QEMU's RPi4 emulation. QEMU is unable to use KVM when emulating a Raspberry Pi 4B because the CPU in the RPi supports EL3, which can be emulated but not virtualized using KVM and QEMU. https://gitlab.com/qemu-project/qemu/-/issues/1073 has some more info on this. It is possible to boot a generated Raspberry Pi image using a more VM-friendly machine type, but it doesn't result in as accurate of Raspberry Pi emulation, which might be undesirable.

Should I use the faster but less accurate mode, or the slower, more accurate one?

@ArrayBolt3
Copy link
Contributor Author

I'll go ahead and implement this using the faster but less accurate virtualization. It's all that's necessary to make sure the image was built properly, the accurate mode is probably only necessary for making sure the kernel or firmware itself is functional on the official hardware itself (which isn't what we need to test, that should be being tested by other projects).

@ArrayBolt3
Copy link
Contributor Author

Raspberry PI image CI testing added. https://github.com/ArrayBolt3/grml-debootstrap/actions/runs/14696085489 will show the results once the tests are done running. (I got them to pass fully in a separate branch first.)

@ArrayBolt3
Copy link
Contributor Author

@mika, @zeha, gentle ping, this should be ready for review. RPi image CI tests have been added and are passing.

Copy link
Member

@mika mika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Briefly went over the code except for grml-debootstrap itself (had to explicitly load the diff to even see it for code review, that was unexpected :-/). I want to clarify the --imagesize vs. --vmsize situation before looking further into the actual grml-debootstrap code changes.

What's definitely missing yet before we can submit this, are according doc changes in grml-debootstrap.8.txt :)

(Thanks for working on this!)

--vmfile \
--vmsize 3G \
${extra_buildopts} \
--imagesize 3G \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, is there a good reason why we couldn't continue using --vmsize and need to replace it with --imagesize? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could keep using --vmsize for sure, but I wanted to switch to --imagesize since now that the argument can be used for more than just VM images, I marked --vmsize as deprecated. They mean the same thing, I figured using the newer syntax would be less confusing, especially since this build command might be building an RPi image, not strictly a VM image (though it will later boot that not-really-a-VM-image in a VM so I guess it doesn't really matter that much).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I'm not yet sure about this change, while I agree that consistency is important and if we keep it backwards compatible at least for now that should be fine. @zeha what's your take on the --vmfile vs --imagesize situation? 🤔

(BTW, AFAICS if someone uses VMSIZE=4G as environment variable on the command line, then this is no longer accepted due to the [ -n "$VMSIZE" ] || VMSIZE="2G" removal, nor?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the environment variable issue. Will leave this thread open so zeha can comment on the --vmsize/--imagesize stuff.

@ArrayBolt3
Copy link
Contributor Author

While working on the docs, I'm realizing the --vm mode (which the --rpi mode is modeled after) automatically enables --defaultinterfaces. I think this makes a lot of sense for Raspberry Pi images. I'll try to sneak that in there before this is merged.

@ArrayBolt3
Copy link
Contributor Author

Tested the new default interfaces commit on my RPi4, works like a charm, don't need to do anything special for the network to work out of the box. CI passes and documentation is updated.

@ArrayBolt3
Copy link
Contributor Author

ArrayBolt3 commented May 4, 2025 via email

@ArrayBolt3
Copy link
Contributor Author

Reworked, tested locally, VMSIZE and IMAGESIZE environment variables are being respected as appropriate now.

@mika
Copy link
Member

mika commented May 14, 2025

Thanks, LGTM. I think we should be good to apply this, though I'm afraid it's nothing we can get into Debian/trixie at this stage. To not block ourselves for further changes/fixes of grml-debootstrap that are supposed to go into Debian/trixie, we need to postpone this merge until the trixie freeze is gone.

Assigning this to our 2025/Q3 project/milestone accordingly, any further testing/feedback appreciated in the meanwhile! :)

@ArrayBolt3
Copy link
Contributor Author

Just rebased this on the current git master. Looks like CI passes: https://github.com/ArrayBolt3/grml-debootstrap/actions/runs/20687641909 Currently testing a cross-build from amd64, will also test "build on RPi, booted on RPi" and see what happens.

@mika
Copy link
Member

mika commented Jan 4, 2026

Just rebased this on the current git master. Looks like CI passes: https://github.com/ArrayBolt3/grml-debootstrap/actions/runs/20687641909 Currently testing a cross-build from amd64, will also test "build on RPi, booted on RPi" and see what happens.

Thanks, please ping us once you think it's ready for review @ArrayBolt3

Our package-build / build debian action currently always seems to fail with:

RequestError [HttpError]: Resource not accessible by integration

Is that a GH issue or do we need to adjust/fix something, @zeha? 🤔

@ArrayBolt3
Copy link
Contributor Author

ArrayBolt3 commented Jan 4, 2026

@mika, @zeha I believe this is ready for review. Both an amd64 cross-build (installing directly to an SD card) and an arm64 native build (installing directly to a USB drive) worked. Both installations were bootable on a Raspberry Pi 4B (and the USB native installation was done from within the amd64 cross-build installation).

Copy link
Member

@zeha zeha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from a glance it looks ok to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Rasberry PI (RPI) support

4 participants