Better isolate new chroot from host /dev, /dev/pts, /run/udev #347

zeha · 2025-06-25T19:34:21Z

Closes: #1108311

Thanks: Wolfgang Zarre

Closes: #1108311 Thanks: Wolfgang Zarre <[email protected]>

zeha · 2025-06-25T22:19:30Z

this breaks grub-install. the default command line ends up being: root=/dev/mapper/loop0p2 which is clearly wrong

fixing it manually in a rescue shell (only /, /dev mounted):

root@(none):/# fdisk -l /dev/vda
Disk /dev/vda: 3 GiB, 3221225472 bytes, 6291456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 1 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4138D79D-B70C-402F-9EE9-4490C5C8F3DE

Device      Start     End Sectors  Size Type
/dev/vda1    2048  206847  204800  100M EFI System
/dev/vda2  206848 6289407 6082560  2.9G Linux filesystem
root@(none):/# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.12.32-arm64
Found initrd image: /boot/initrd.img-6.12.32-arm64
Adding boot menu entry for UEFI Firmware Settings ...
done
root@(none):/# cat /boot/grub/grub.cfg  | grep root
search --no-floppy --fs-uuid --set=root ee7bde03-4fe9-496d-bb04-090a3d57045b
        search --no-floppy --fs-uuid --set=root ee7bde03-4fe9-496d-bb04-090a3d57045b
        linux   /boot/vmlinuz-6.12.32-arm64 root=UUID=ee7bde03-4fe9-496d-bb04-090a3d57045b ro  quiet console=ttyS0,115200 console=tty0 vga=791
                search --no-floppy --fs-uuid --set=root ee7bde03-4fe9-496d-bb04-090a3d57045b
                linux   /boot/vmlinuz-6.12.32-arm64 root=UUID=ee7bde03-4fe9-496d-bb04-090a3d57045b ro  quiet console=ttyS0,115200 console=tty0 vga=791
                search --no-floppy --fs-uuid --set=root ee7bde03-4fe9-496d-bb04-090a3d57045b
                linux   /boot/vmlinuz-6.12.32-arm64 root=UUID=ee7bde03-4fe9-496d-bb04-090a3d57045b ro single

zeha · 2025-06-25T23:09:15Z

failed build:

+ grub_targets=/dev/loop0
+ local device
+ for device in $grub_targets
+ echo 'Installing grub on /dev/loop0:'
+ '[' -n /dev/mapper/loop0p1 ']'
+ case "${ARCH}" in
+ run_grub_install --no-floppy --target=arm64-efi --force-extra-removable /dev/loop0
+ grub_install_cmd=('grub-install' "$@")
+ grub-install --no-floppy --target=arm64-efi --force-extra-removable /dev/loop0
Installing for arm64-efi platform.
Installation finished. No error reported.
+ '[' grub-efi-arm64 = grub-cloud-amd64 ']'
+ '[' -z /dev/mapper/loop0p1 ']'
+ rm -f /boot/grub/device.map
+ '[' -n 'console=ttyS0,115200 console=tty0 vga=791' ']'
+ echo 'Adding BOOT_APPEND configuration ['\''console=ttyS0,115200 console=tty0 vga=791'\''] to /etc/default/grub.'
+ sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/ s#"$# console=ttyS0,115200 console=tty0 vga=791"#' /etc/default/grub
Adding BOOT_APPEND configuration ['console=ttyS0,115200 console=tty0 vga=791'] to /etc/default/grub.
+ update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-37-arm64
Found initrd image: /boot/initrd.img-6.1.0-37-arm64
Adding boot menu entry for UEFI Firmware Settings ...
done
XXXXXXXXXXXXX mount output follows
+ echo 'XXXXXXXXXXXXX mount output follows'
+ mount
/dev/mapper/loop0p2 on / type ext4 (rw,relatime)
tmpfs on /dev type tmpfs (ro,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
none on /proc type proc (rw,relatime)
none on /sys type sysfs (rw,relatime)
/dev/mapper/loop0p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,relatime)
XXXXXXXXXXXXX grub.cfg follows
+ echo 'XXXXXXXXXXXXX grub.cfg follows'
+ cat /boot/grub/grub.cfg
+ echo 'XXXXXXXXXXXXX end'
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
+ mountpoint -q /boot/efi
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi
function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}
if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 619677a0-9fd3-49c8-b5fd-a239b2d2465b
    font="/usr/share/grub/unicode.pf2"
fi
if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=5
  fi
fi
### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###
### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-619677a0-9fd3-49c8-b5fd-a239b2d2465b' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root 619677a0-9fd3-49c8-b5fd-a239b2d2465b
	echo	'Loading Linux 6.1.0-37-arm64 ...'
	linux	/boot/vmlinuz-6.1.0-37-arm64 root=/dev/mapper/loop0p2 ro  quiet console=ttyS0,115200 console=tty0 vga=791
	echo	'Loading initial ramdisk ...'

zeha · 2025-06-26T00:43:48Z

good build:

+ run_grub_install --no-floppy --target=arm64-efi --force-extra-removable /dev/loop0
+ grub_install_cmd=('grub-install' "$@")
+ grub-install --no-floppy --target=arm64-efi --force-extra-removable /dev/loop0
Installing for arm64-efi platform.
Installation finished. No error reported.
+ '[' grub-efi-arm64 = grub-cloud-amd64 ']'
+ '[' -z /dev/mapper/loop0p1 ']'
+ rm -f /boot/grub/device.map
+ '[' -n 'console=ttyS0,115200 console=tty0 vga=791' ']'
Adding BOOT_APPEND configuration ['console=ttyS0,115200 console=tty0 vga=791'] to /etc/default/grub.
+ echo 'Adding BOOT_APPEND configuration ['\''console=ttyS0,115200 console=tty0 vga=791'\''] to /etc/default/grub.'
+ sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/ s#"$# console=ttyS0,115200 console=tty0 vga=791"#' /etc/default/grub
+ update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-37-arm64
Found initrd image: /boot/initrd.img-6.1.0-37-arm64
Adding boot menu entry for UEFI Firmware Settings ...
done
XXXXXXXXXXXXX mount output follows
+ echo 'XXXXXXXXXXXXX mount output follows'
+ mount
/dev/mapper/loop0p2 on / type ext4 (rw,relatime)
udev on /dev type devtmpfs (rw,relatime,size=8147432k,nr_inodes=2036858,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
none on /proc type proc (rw,relatime)
none on /sys type sysfs (rw,relatime)
/dev/mapper/loop0p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,relatime)
+ echo 'XXXXXXXXXXXXX grub.cfg follows'
+ cat /boot/grub/grub.cfg
XXXXXXXXXXXXX grub.cfg follows
#
+ echo 'XXXXXXXXXXXXX end'
+ mountpoint -q /boot/efi
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root d0462186-3859-428e-9cfd-0e6e56c70e0b
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=5
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-d0462186-3859-428e-9cfd-0e6e56c70e0b' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root d0462186-3859-428e-9cfd-0e6e56c70e0b
	echo	'Loading Linux 6.1.0-37-arm64 ...'
	linux	/boot/vmlinuz-6.1.0-37-arm64 root=UUID=d0462186-3859-428e-9cfd-0e6e56c70e0b ro  quiet console=ttyS0,115200 console=tty0 vga=791
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-6.1.0-37-arm64
}
submenu 'Advanced options for Debian GNU/Linux' $menuentry_id_option 'gnulinux-advanced-d0462186-3859-428e-9cfd-0e6e56c70e0b' {
	menuentry 'Debian GNU/Linux, with Linux 6.1.0-37-arm64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.1.0-37-arm64-advanced-d0462186-3859-428e-9cfd-0e6e56c70e0b' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		search --no-floppy --fs-uuid --set=root d0462186-3859-428e-9cfd-0e6e56c70e0b
		echo	'Loading Linux 6.1.0-37-arm64 ...'
		linux	/boot/vmlinuz-6.1.0-37-arm64 root=UUID=d0462186-3859-428e-9cfd-0e6e56c70e0b ro  quiet console=ttyS0,115200 console=tty0 vga=791

zeha · 2025-06-26T00:48:18Z

quick thought: /dev/mapper might be a userspace construct? but i guess the host udev should still deal with it...

mika · 2025-06-29T14:23:08Z

JFTR, forwarding comment from Wolfgang Zarre at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1108311#24:

Maybe I should have mentioned, that I bootstraped with grub disable to a mounted image
file with UKI images for a Secure Boot environment, which can run in VM and on real
hardware.

I'll have a closer look at this issue next week because I'm really interested in solving
this issue due to the fact that I use grml-debootstrap a lot and maybe sometime in the
future even with a grub loader.

I'll inform you as soon as I have an idea.

zeha · 2025-06-29T20:20:08Z

I've asked around a bit and the consensus is: linux devtmpfs will not help us. What we probably need to do is create our own /dev, and use udev to populate it before installing the boot loader etc.

zeha · 2025-06-29T20:54:10Z

Related: #348

the-linux-wolf · 2025-07-04T06:43:19Z

Ok, joined now the party here.
Hmmm, very interesting, I cannot reproduce the mentioned failure above on my systems.
I bootstrapped several EFI based images with zeha/1108311 which are having a valid grub.cfg with correct UUID's and those are booting normal in qemu.
Seems to be rather another issue here, will have a look at it.

the-linux-wolf · 2025-07-05T17:56:15Z

Alright, issue found, as suspected, the issue is with the build environment and not really
the changes made in grml-debootstrap.

The issue is simply that docker mounts /dev by default with type tmpfs, which then lacks
device files compared to the mount type devtmpfs.

Missing device files are then created during runtime, also in chroot, but obviously that will not
work, when /dev is mounted read-only and therefore grub.cfg is not created correctly.

To make this work now, we can add another option to the docker run command, which just simply is
a bind mount of /dev as it is on the host.

However, in the container we also have to set the bind mount to read-only, due to the fact that docker
is suffering from the very same issue with /dev , which could modify the hosts /dev during
the setup of the container.

Inside of the container it would look like this:

# findmnt /dev
TARGET SOURCE FSTYPE   OPTIONS
/dev   udev   devtmpfs ro,nosuid,relatime,size=30740364k,nr_inodes=7685091,mode=755,inode64

That will do the trick, and all steps as according to tests/README.md will succeed with the
following diff:

diff --git a/tests/build-vm-and-test.sh b/tests/build-vm-and-test.sh
index b7d7b56..895618f 100755
--- a/tests/build-vm-and-test.sh
+++ b/tests/build-vm-and-test.sh
@@ -64,6 +64,7 @@ if [ "$1" == "run" ]; then

    # we need to run in privileged mode to be able to use loop devices
    exec docker run --privileged --rm -i \
+    --mount type=bind,src=/dev,dst=/dev,ro=true \
      -v "$(pwd)":/code \
      -e TERM="$TERM" \
      -w /code \

This is another proof that docker is not the miracle cure for development and rather suboptimal,
especially for system or system near development.

On the other hand qemu is already used here and therefore it could be used for the build as well,
which would be more reliable, independent to the hosts system and less of a hassle.

zeha self-assigned this Jun 25, 2025

zeha force-pushed the zeha/1108311 branch from c0119d2 to 07e9fec Compare June 25, 2025 19:38

Better isolate new chroot from host /dev, /dev/pts, /run/udev

074a2c1

Closes: #1108311 Thanks: Wolfgang Zarre <[email protected]>

zeha force-pushed the zeha/1108311 branch from 07e9fec to 074a2c1 Compare June 25, 2025 19:39

debug

f9eb514

zeha force-pushed the zeha/1108311 branch from 9d17d6b to f9eb514 Compare June 25, 2025 22:39

zeha force-pushed the zeha/1108311 branch from ee55eb5 to f9eb514 Compare July 3, 2025 10:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Better isolate new chroot from host /dev, /dev/pts, /run/udev #347

Better isolate new chroot from host /dev, /dev/pts, /run/udev #347

Uh oh!

zeha commented Jun 25, 2025 •

edited

Loading

Uh oh!

zeha commented Jun 25, 2025 •

edited

Loading

Uh oh!

zeha commented Jun 25, 2025

Uh oh!

zeha commented Jun 26, 2025

Uh oh!

zeha commented Jun 26, 2025

Uh oh!

mika commented Jun 29, 2025

Uh oh!

zeha commented Jun 29, 2025

Uh oh!

zeha commented Jun 29, 2025

Uh oh!

the-linux-wolf commented Jul 4, 2025

Uh oh!

the-linux-wolf commented Jul 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Better isolate new chroot from host /dev, /dev/pts, /run/udev #347

Are you sure you want to change the base?

Better isolate new chroot from host /dev, /dev/pts, /run/udev #347

Uh oh!

Conversation

zeha commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeha commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeha commented Jun 25, 2025

Uh oh!

zeha commented Jun 26, 2025

Uh oh!

zeha commented Jun 26, 2025

Uh oh!

mika commented Jun 29, 2025

Uh oh!

zeha commented Jun 29, 2025

Uh oh!

zeha commented Jun 29, 2025

Uh oh!

the-linux-wolf commented Jul 4, 2025

Uh oh!

the-linux-wolf commented Jul 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zeha commented Jun 25, 2025 •

edited

Loading

zeha commented Jun 25, 2025 •

edited

Loading