to-filesystem with --source-imgref

[cgwalters]

[anaconda root@fedora ~]# bootc install to-disk --source-imgref containers-storage:quay.io/centos-bootc/centos-bootc:stream9 --stateroot default --filesystem ext4 /dev/vda --wipe
Installing image: docker://quay.io/centos-bootc/centos-bootc:stream9
Wiping /dev/vda1
Wiping device /dev/vda1
Wiping /dev/vda2
Wiping device /dev/vda2
/dev/vda2: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
Wiping /dev/vda3
Wiping device /dev/vda3
/dev/vda3: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
Wiping /dev/vda
Wiping device /dev/vda
/dev/vda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/vda: 8 bytes were erased at offset 0x4fffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/vda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/vda: calling ioctl to re-read partition table: Success
Block setup: direct
       Size: 21474836480
     Serial: <unknown>
      Model: <unknown>
ERROR Installing to disk: Creating rootfs: Loading SELinux policy: SELinux enabled, but no policy found in root

This is for work on Anaconda. Today, bootc install defaults to assuming it's run from the container image being installed. --source-imgref turns this off, and it's used by bootc-image-builder today. However, I'd really like going forward in general to lean strongly into the "run bootc install from target container" as much as possible, and this would mean that Anaconda would need to fetch (into RAM) the target container image by default, then run it. I know this would be a pretty substantial change, but it would have a lot of benefits.

But to continue with the current architecture, when using --source-imgref today that turns off the "install self container" mode, and it does mean that some infrastructure (such as selinux-policy) to bootstrap its labeling state.

I think we could probably lift this requirement and fall back to doing labeling as a secondary post-fixup path, using the policy from the target root. That would be a bit involved but doable.

[travier]

Another path is to add Anaconda to an existing bootc image and then generate a LiveISO out of it. It's currently blocked on:

• Add support for bootc rhinstaller/anaconda#5197
• Add support for iso osbuild/bootc-image-builder#433

[elkoniu]

Today I have made some more experients and faced two issues.
First - bootc is looking for a file etc/ostree/prepare-root.conf which is not presented by default. It is just enough to create empty file for bootc to move forward.
Q1) What is the purpose of this file and why bootc needs it? If this is some missing libostree dependency?

Then I have an empty partition mounted with just empty /boot directory inside. This time bootc complains about missing selinux policies:

[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 --target-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 /mnt/sysroot
Installing image: docker://containers-storage:quay.io/centos-bootc/centos-bootc:stream9
ERROR Installing to filesystem: Creating ostree deployment: Loading SELinux policy: SELinux enabled, but no policy found in root

Q2) If this is related to empty prepare-root.conf file or just some basic filesystem policy is missing?

[jlebon]

What is the purpose of this file and why bootc needs it?

It looks like it uses it to know if to turn on fs-verity on the repo:

bootc/lib/src/install.rs

Line 684 in 5ed53c3

if prepare_root_composefs.requires_fsverity() || repo_verity_state.desired == Tristate::Enabled

But yeah, I think the main gap here is that one would expect that with --source-imgref, the prepare-root.conf to check comes from the target image and not the host environment. The problem of course is that at that point, the container image hasn't been pulled yet (because the filesystem we're pulling it into is still being initialized!).

So... one seemingly obvious fix here is to delay turning on fs-verity until after we deploy the container image? I think the main downside of that is that it's much more expensive because you now have a lot more objects to iterate over. It's the same amount of I/O (I think) total, but when done upfront it's amortized over the whole ostree import operation. And obviously, we could do this only in the --source-imgref case.

[cgwalters]

➡️ #1410

[jkonecny12]

I definitely see the benefits of running bootc from inside the container and I like the idea. However, it is just not possible to make this default as size of the container images might be quite huge. The nice example are supported RHEL AI containers which has over 100GB size IIRC. We can't run these containers from a memory.

If we want to go this way we would have to have some way of doing the partitioning first and then use that to download the container and install from that. We already doing something like that with DNF transaction. I think this might be doable but we would need to make this coexist with the bootc installation. I mean, bootc needs to allow us to use some of the space for the container image and not erase that during installation.

[cgwalters]

I definitely see the benefits of running bootc from inside the container and I like the idea. However, it is just not possible to make this default as size of the container images might be quite huge. The nice example are supported RHEL AI containers which has over 100GB size IIRC. We can't run these containers from a memory.

I suggested this in rhinstaller/anaconda#5197 (comment) :

One option I think would be viable is to always create a temporary swap file on that filesystem.

That would involve double I/O, but would mean we weren't limited to RAM.

As far as running from memory though, remember for RHELAI a primary target install is bare metal servers which can easily have more than 100G of RAM. Over 1TB is not at all uncommon.