Composefs-native backend #1314

Johan-Liebert1 · 2025-05-13T07:47:00Z

Related to: #1190

A draft for installing and booting using composefs-rs.

Works with install to-disk with --composefs-native option
Works with install to-filesystem with --composefs-native option
Implementation for bootc status
Implementation for bootc switch, bootc upgrade and bootc rollback. Some edge cases need handling, such as rolling back while a deployment is staged. Currently staged deployment WILL be booted on the next reboot regardless of rolling back.

Issues:

Issue with transport type continers-storage, viz, #117. Needs some ironing out. Done
Bootc images need rebuilding with composfs-rs components, namely composfs-setup-root

I have a few logs being logged as warning, just for visibility. Will remove them when it's ready.
Same with composefs-rs patch in cargo.toml

cgwalters

Thanks for working on this!!

cgwalters · 2025-05-13T10:23:04Z

lib/src/install.rs

+        .create_dir_all("composefs")
+        .context("Creating dir 'composefs'")?;
+
+    tracing::warn!("STATE: {state:#?}");


I'd recommend dbg! for this type of stuff; it will build locally but not pass CI so it can't be accidentally merged.

I intend to remove all these logs. They were only for debugging purposes

Yep that makes sense, what I meant is that use dbg! locally. (Or use tracing::debug! and keep the tracing in there and just use the tracing selector to ensure you see what you want...we are really chatty with a big env RUST_LOG=debug unfortunately today, mostly in container fetches)

cgwalters · 2025-05-13T10:23:21Z

lib/src/cli.rs

@@ -1199,7 +1199,7 @@ async fn run_from_opt(opt: Opt) -> Result<()> {
                    let fd =
                        std::fs::File::open(&path).with_context(|| format!("Reading {path}"))?;
                    let digest: fsverity::Sha256HashValue = fsverity::measure_verity(&fd)?;
-                    let digest = hex::encode(digest);
+                    let digest = digest.to_hex();


I think this should be squashed with the previous commit, and it may make sense to have a "bump composefs-rs" prep PR split out right?

Right, that makes sense. I'll put this in a separate PR

Done in #1403

cgwalters · 2025-05-13T10:24:46Z

lib/src/install.rs

 async fn install_to_filesystem_impl(
    state: &State,
    rootfs: &mut RootSetup,
    cleanup: Cleanup,
+    composefs: bool,


I think this would make sense as part of state

I was conflicted on where to put this. Thanks for the review. I'll add this to State

cgwalters · 2025-05-13T11:14:46Z

lib/src/install.rs

-            &imgstore,
-        )
-        .await?;
+        tracing::warn!(


Same comment re dbg! or this one could also make sense as a tracing::debug! that gets left in right?

Ah, yes. We can keep this one as debug

cgwalters · 2025-05-13T13:07:15Z

bootc install to-disk --composefs

The branding here is a bit confusing because of course, bootc already uses composefs. Just the https://github.com/composefs/composefs/ version, and that is "indirectly" via ostree.

Bigger picture I think what we're trying to aim for here primarily is "sealed" images right? In that case then, one thing I think would make sense is that we don't even have an option to install - we determine what to do by inspecting the container.

It does of course make sense to eventually support unsealed images this way too...which will be extremely important for eventually migrating existing ostree systems. There's an enormous set of details there though.

So my proposal in a nutshell:

Let's call it "composefs native" storage or so (the branding here is again hard, open to other ideas)
Default to (i.e. require) composefs-native storage if we detect the image is a sealed UKI.
Make the opt-in flag called like --composefs-native or so for unsealed cases. That said, it could also make sense to add some sort of flag to the container image to do this instead; because I think most people who want to do this will always want to do it. It's basically similar to what we have with kernel arguments; they can be either baked into the container or applied out of band, and in the general case we need both.

Johan-Liebert1 · 2025-05-15T04:49:31Z

Let's call it "composefs native" storage or so (the branding here is again hard, open to other ideas)

I think this works. Or we could have --composefs-native-experimental or something?

Default to (i.e. require) composefs-native storage if we detect the image is a sealed UKI.

Curious about this. Is there a reason we'd want composfs native storage in this case?

cgwalters · 2025-05-15T13:11:20Z

Curious about this. Is there a reason we'd want composfs native storage in this case?

Isn't the whole idea that we're aiming for the sealed case?

Johan-Liebert1 · 2025-06-05T08:44:24Z

I'll be getting back to this PR next Monday. Been working on a separate issue this week...

cgwalters · 2025-06-16T18:30:04Z

lib/src/install/baseline.rs

@@ -403,6 +409,8 @@ pub(crate) fn install_create_rootfs(
        "root",
        opts.wipe,
        mkfs_options.iter().copied(),
+        // TODO: Add cli option for this
+        Some(uuid::uuid!("6523f8ae-3eb1-4e2a-a05a-18b695ae656f")),


It looks like you chose the alpha architecture uuid; it'd make sense potentially to use the DPS UUIDs by default for to-disk but that will require some work.

cc #448 (comment)

cgwalters · 2025-06-25T15:47:10Z

lib/src/install.rs

+    let esp_devices = [COREOS_ESP_PART_LABEL, ANACONDA_ESP_PART_LABEL]
+        .into_iter()
+        .map(|p| Path::new("/dev/disk/by-partlabel/").join(p));


Two things

We shouldn't use /dev/disk/by-partlabel because that's global state, in the general case we may be creating partitions on a loopback mounted device and don't want to touch any global state

There's a well known UUID for the ESP to use

Combining these, see https://github.com/coreos/bootupd/blob/92a8fe2326e35d893afbe464857408c403d9e7ea/src/blockdev.rs#L41 which perhaps we could move into this codebase and have bootupd use it (because bootupd already uses bootc).

Ah, okay I'll update to not use by-partlabel

perhaps we could move into this codebase and have bootupd use it (because bootupd already uses bootc)

I found this little bit weird, as I feel like this should be the other way around. Is there a reason why bootc uses the bootupctl command rather than the bootup crate?

Is there a reason why bootc uses the bootupctl command rather than the bootup crate?

Well...unless we took over that CLI, that'd mean we duplicate the code.

cgwalters · 2025-06-25T15:47:42Z

lib/src/install.rs

+    create_dir_all(&boot_dir).context("Failed to create boot dir")?;
+
+    // Add the user grug cfg
+    let grub_user_config = format!(


In my ideal world grub related things live in bootupd, not in this project.

And even better if we can make this a static configuration...where grub basically implements the BLS spec and finds things in /EFI/Linux on its own right?

It makes sense to put this in bootupd, but upon discussions with @travier we're still debating on whether we want to push systemd-boot or not, so this is more or less temporary

This part of the GRUB config can not be generated by bootup as it is the deployment boot order in the UKI + GRUB case. See: containers/composefs-rs#38 (comment)

It's not possible to use the BLS spec partial support in GRUB to boot UKI. It does not support the key that we need to do so, thus why we generate GRUB config snippets: ostreedev/ostree#2753 (comment)

cgwalters

Some more comments

lib/Cargo.toml

ostree-ext/Cargo.toml

ostree-ext/src/container_utils.rs

cgwalters · 2025-07-16T12:20:26Z

ostree-ext/src/container_utils.rs

+
+/// Returns true if the system appears to have been booted with composefs.
+pub fn composefs_booted() -> io::Result<bool> {
+    let cmdline = std::fs::read_to_string("/proc/cmdline")?;


We should use a proper kernel argument parser in general. It's actually a bit complicated because of quoting. systemd has a good version of course, and we have one in C in ostree. It'd probably make sense to make a dedicated one.

We could build our own parser? @travier suggested https://github.com/rust-bakery/nom for parsing Grub menuentries. Maybe we could use it for parsing kernel cmdline as well?

I have written what feels like a million various nom parsers for Advent of Code over the years, I could probably throw something together for kernel cmdline in short order. Seems to be well-documented at https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

Sure, I'm good to make a new small internal crate for this. But I wouldn't try to just greenfield it though, but reference existing parsers. I think https://github.com/ostreedev/ostree/blob/main/src/libostree/ostree-kernel-args.c is correct, as is the systemd version.

Sorry I forgot we already have https://github.com/bootc-dev/bootc/blob/main/lib/src/kernel.rs and to combine with the chat yes let's for now just take the code from https://github.com/containers/composefs-rs/blob/d5455222f282200913902f29c66d5f625ca5661f/crates/composefs-boot/src/cmdline.rs#L9 but what we IMO must do when parsing /proc/cmdline is use to_utf8_lossy or so so we don't bomb out if the kernel commandline somehow has non-UTF8 data.

jeckersb · 2025-07-16T20:06:08Z

lib/src/bls_config.rs

👍 for a first pass, but skimming through the boot loader specification document there's a lot more to this. Probably enough so that it would make sense to spin this off into its own crate, I'm sure others would find it to be useful outside of bootc.

Probably enough so that it would make sense to spin this off into its own crate,

Yes I had the same thought, was one of the motivations for #1413

I'm sure others would find it to be useful outside of bootc.

Probably though I'd say we should probably not at the current time try to go all the way into making it a real published crate unless we had a reason to believe that there was another possible concrete user.

cgwalters

Some more review

lib/src/install.rs

lib/src/deploy.rs

lib/src/cli.rs

cgwalters · 2025-07-16T21:03:58Z

lib/src/install.rs

+async fn initialize_composefs_repository(
+    state: &State,
+    root_setup: &RootSetup,
+) -> Result<(Sha256Digest, impl FsVerityHashValue)> {


I think we should return the repository object too.

Yeah, I missed it

lib/src/install.rs

cgwalters · 2025-07-16T21:15:30Z

lib/src/install.rs

+#[context("Setting up BLS boot")]
+pub(crate) fn setup_composefs_bls_boot(
+    setup_type: BootSetupType,
+    // TODO: Make this generic


Yeah but you know the problem here is...it will basically infect most of the bootc codebase with a generic - the same way it did in composefs-rs. That's going to be super painful.

We had part of this debate in containers/composefs-rs#105 and I'm still kind of skeptical that generics make sense overall. (The especially important argument here is that we may want to defer until runtime i.e. via some repository configuration or other state which digest we use) and that will mean we basically double 70% (yes number made up) of the code generated.

A potential thing in the future is to support migrating from sha256 to sha512 for example, and then we'd need two instantations of Repository for each.

containers/composefs-rs#159

lib/src/install.rs

jeckersb · 2025-07-17T17:31:48Z

lib/src/bls_config.rs

+    }
+}
+
+pub(crate) fn parse_bls_config(input: &str) -> Result<BLSConfig> {


This sort of thing I would normally impl TryFrom<&str> for BLSConfig. I think it makes it more obvious what is going on when you use it like let foo = BLSConfig::try_from(&some_str)?; Plus for free you also get the inverse of let foo: BLSConfig = some_str.try_into()?;

cgwalters · 2025-07-21T12:22:58Z

We had a live chat about this and decided to make a branch in this repo for it so multiple people can collaborate!

cgwalters · 2025-07-21T12:46:59Z

Created in https://github.com/bootc-dev/bootc/tree/composefs-backend
I have also sent an invite to @Johan-Liebert1 to have write access to the repository.

(That said we still need to convert to e.g. https://clowarden.io/ )

Instead of using `write_boot_simple` from composefs-rs, have custom logic in bootc to write UKIs and Grub menuentries Signed-off-by: Johan-Liebert1 <[email protected]>

Signed-off-by: Johan-Liebert1 <[email protected]>

Use `atomic_write` from `cap_std` crate Get rid of `opeanat` crate Call `fsync` after write to disk Signed-off-by: Johan-Liebert1 <[email protected]>

crates/lib/src/install.rs

crates/ostree-ext/Cargo.toml

crates/ostree-ext/src/container_utils.rs

Remove `--boot` option as we can get it from the image itself. Allow `--insecure` option to `--composefs-native` to make fsverity validation optional in case the filesystem does not support it. Signed-off-by: Johan-Liebert1 <[email protected]>

Signed-off-by: Johan-Liebert1 <[email protected]>

Parse the Grub menuentry file, `boot/grub2/user.cfg` to get a list of bootable UKIs and figure out if a rollback is currently queued. Signed-off-by: Johan-Liebert1 <[email protected]>

Returning a local reference to a `&str` is quite tricky with rust. Update `title` and `chainloader`, the two dynamic fields in the grub menuentry, to be `String` instead of `&str` Signed-off-by: Johan-Liebert1 <[email protected]>

We parse the grub menuentries, get the rollback deployment then perform the rollback, which basically consists of writing a new .staged menuentry file then atomically swapping the staged and the current menuentry. Rollback while there is a staged deployment is still to be handled. Signed-off-by: Johan-Liebert1 <[email protected]>

…iles Signed-off-by: Johan-Liebert1 <[email protected]>

If two deployments have the same VMLinuz + Initrd then, we can use the same binaries for both the deployments. Before writing the BLS entries to disk we calculate the SHA256Sum of VMLinuz + Initrd combo, then test if any other deployment has the same SHA256Sum for the binaries. Store the hash in the origin file under `boot -> hash` for future lookups. Signed-off-by: Johan-Liebert1 <[email protected]>

Signed-off-by: Johan-Liebert1 <[email protected]>

Centralize all constants in a separate file Signed-off-by: Johan-Liebert1 <[email protected]>

Instead of `/sysroot/state/os/fedora` use `/sysroot/state/os/default` as the default state directory. Signed-off-by: Johan-Liebert1 <[email protected]>

Signed-off-by: Johan-Liebert1 <[email protected]>

Instaed of writing all present menuentries, only write the menuentry for switch/upgrade and the menuentry for the currently booted deployment. Signed-off-by: Johan-Liebert1 <[email protected]>

Signed-off-by: Johan-Liebert1 <[email protected]>

This allows for easier testing Signed-off-by: Pragyan Poudyal <[email protected]>

Add tests for functions `get_sorted_bls_boot_entries` and `get_sorted_uki_boot_entries` Signed-off-by: Pragyan Poudyal <[email protected]>

cgwalters · 2025-08-01T12:36:04Z

Two things:

@Johan-Liebert1 where are we on this PR vs the branch on main WIP: composefs branch #1444 ?
Can you (or someone) review store: Add support for composefs-rs #1471 ? I want to try rebasing the branch on that
Also in the vein of rebasing I think when we do that we can probably do some squashing and also try to factor out some prep commits (maybe the GRUB parser?)

cgwalters

Overall looks sane, we will definitely have some splitting/squashing/cleanup/testing to do as we target main though

github-actions bot added the area/install Issues related to `bootc install` label May 13, 2025

Johan-Liebert1 force-pushed the composefs-backend branch from cd42079 to b3d1952 Compare May 13, 2025 09:08

cgwalters reviewed May 13, 2025

View reviewed changes

cgwalters mentioned this pull request May 13, 2025

examples: add bootc/ directory containers/composefs-rs#116

Draft

travier mentioned this pull request Jun 16, 2025

examples: Add bootc UKI & BLS examples containers/composefs-rs#143

Draft

cgwalters assigned jeckersb Jun 16, 2025

cgwalters requested a review from jeckersb June 16, 2025 18:24

cgwalters reviewed Jun 16, 2025

View reviewed changes

Johan-Liebert1 force-pushed the composefs-backend branch from 5979f04 to 42632ce Compare June 17, 2025 05:59

travier mentioned this pull request Jun 17, 2025

bootc composefs-native backend #1190

Open

12 tasks

cgwalters changed the title ~~Composefs backend~~ Composefs-native backend Jun 25, 2025

cgwalters mentioned this pull request Jun 25, 2025

Investigate alignment/intersection with systemd-repart #448

Open

cgwalters reviewed Jun 25, 2025

View reviewed changes

Johan-Liebert1 force-pushed the composefs-backend branch from 25602da to 7dcc0fc Compare July 14, 2025 16:27

cgwalters requested changes Jul 16, 2025

View reviewed changes

jeckersb reviewed Jul 16, 2025

View reviewed changes

cgwalters reviewed Jul 16, 2025

View reviewed changes

jeckersb reviewed Jul 17, 2025

View reviewed changes

This was referenced Jul 17, 2025

Clean up kernel argument parsing #1425

Closed

Mitigate generics containers/composefs-rs#159

Open

Johan-Liebert1 force-pushed the composefs-backend branch from 7dcc0fc to 04f048f Compare July 19, 2025 11:16

Johan-Liebert1 added 3 commits July 23, 2025 10:42

install/composefs: Move UKI write logic to bootc

8b56e8f

Instead of using `write_boot_simple` from composefs-rs, have custom logic in bootc to write UKIs and Grub menuentries Signed-off-by: Johan-Liebert1 <[email protected]>

install: Add cmdline kargs to RootSetup

734f43d

Signed-off-by: Johan-Liebert1 <[email protected]>

install/composefs: Use atomic writes

b0e43a8

Use `atomic_write` from `cap_std` crate Get rid of `opeanat` crate Call `fsync` after write to disk Signed-off-by: Johan-Liebert1 <[email protected]>

cgwalters reviewed Jul 24, 2025

View reviewed changes

crates/lib/src/install.rs Outdated Show resolved Hide resolved

crates/ostree-ext/Cargo.toml Outdated Show resolved Hide resolved

crates/ostree-ext/src/container_utils.rs Outdated Show resolved Hide resolved

cgwalters mentioned this pull request Jul 24, 2025

WIP: composefs branch #1444

Draft

Johan-Liebert1 changed the base branch from main to composefs-backend July 24, 2025 13:06

Johan-Liebert1 marked this pull request as ready for review July 29, 2025 11:22

Johan-Liebert1 requested a review from cgwalters July 29, 2025 11:22

Johan-Liebert1 force-pushed the composefs-backend branch 2 times, most recently from f068622 to 72bd34b Compare July 31, 2025 06:15

Johan-Liebert1 added 15 commits August 1, 2025 14:48

cli/composefs: Change composefs options

cf70d5f

Remove `--boot` option as we can get it from the image itself. Allow `--insecure` option to `--composefs-native` to make fsverity validation optional in case the filesystem does not support it. Signed-off-by: Johan-Liebert1 <[email protected]>

Write nom parser for Grub menuentries

ca9307d

Signed-off-by: Johan-Liebert1 <[email protected]>

composefs/status: Read UKI entries to check for queued rollback

c89efab

Parse the Grub menuentry file, `boot/grub2/user.cfg` to get a list of bootable UKIs and figure out if a rollback is currently queued. Signed-off-by: Johan-Liebert1 <[email protected]>

parser/grub: Use String instead of &str

07ad2d3

Returning a local reference to a `&str` is quite tricky with rust. Update `title` and `chainloader`, the two dynamic fields in the grub menuentry, to be `String` instead of `&str` Signed-off-by: Johan-Liebert1 <[email protected]>

composefs/state: Use atomic writes for origin and staged deployment f…

2c63434

…iles Signed-off-by: Johan-Liebert1 <[email protected]>

parser/bls: impl Display for BLSConfig

5870f28

Signed-off-by: Johan-Liebert1 <[email protected]>

lib/composefs: Centralize constants

4ff4098

Centralize all constants in a separate file Signed-off-by: Johan-Liebert1 <[email protected]>

composefs/state: Name state directory default

09d77a5

Instead of `/sysroot/state/os/fedora` use `/sysroot/state/os/default` as the default state directory. Signed-off-by: Johan-Liebert1 <[email protected]>

parser/bls: Add tests for bls parser

4c31ddf

Signed-off-by: Johan-Liebert1 <[email protected]>

install/composefs/uki: Write only staged + booted menuentry on upgrade

1e73eaa

Instaed of writing all present menuentries, only write the menuentry for switch/upgrade and the menuentry for the currently booted deployment. Signed-off-by: Johan-Liebert1 <[email protected]>

rollback/composefs: Print whether we are reverting the queued rollback

782a3d6

Signed-off-by: Johan-Liebert1 <[email protected]>

refactor: Pass boot dir to boot entry readers

f0ffe64

This allows for easier testing Signed-off-by: Pragyan Poudyal <[email protected]>

test: Add tests for reading boot entries

3d2385d

Add tests for functions `get_sorted_bls_boot_entries` and `get_sorted_uki_boot_entries` Signed-off-by: Pragyan Poudyal <[email protected]>

Johan-Liebert1 force-pushed the composefs-backend branch from 72bd34b to 3d2385d Compare August 1, 2025 10:07

cgwalters approved these changes Aug 1, 2025

View reviewed changes

cgwalters merged commit 18c5a1d into bootc-dev:composefs-backend Aug 1, 2025
25 checks passed

Composefs-native backend #1314

Composefs-native backend #1314

Uh oh!

Conversation

Johan-Liebert1 commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented May 13, 2025

Uh oh!

Johan-Liebert1 commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented May 15, 2025

Uh oh!

Johan-Liebert1 commented Jun 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Johan-Liebert1 Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeckersb Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Johan-Liebert1 commented May 13, 2025 •

edited

Loading

Johan-Liebert1 commented May 15, 2025 •

edited

Loading

Johan-Liebert1 Jun 26, 2025 •

edited

Loading

jeckersb Jul 17, 2025 •

edited

Loading

cgwalters commented Aug 1, 2025 •

edited

Loading