RFC: Adding a "light" bootc mode

[jeckersb]

My initial inspiration for something like this came up while thinking about how to better integrate image-based bootc systems with traditional package-based systems. Somewhat similar to what @cgwalters has proposed in #1656 wherein he wants to make it easy to change a bootc system into a package-based system. My proposal here comes at things from a different angle and these ideas are not mutually exclusive, but both live in the space where we want to make it easier to transition between modes of operation.

In my proposal, we would support an installation mode where the system is initially created/provisioned using bootc, using a container image the same as any other bootc system. However, the difference would be that after the initial installation, the system would have a persistent read-write overlay on top of the root.

Some benefits from such an arrangement:

• Installation via anaconda (or $other_installer) would be faster than doing a traditional package-based install. This needs verified since I have not actually tested install speeds of container-based vs package-based anaconda install flows. But it intuitively seems it should be the case if you can do all of the package transaction and scriptlet processing ahead of time that it's faster to lay it out onto disk from a container.
• The system would always have a "factory reset" option which would be activated by discarding the overlay and rebooting the system to return to the initial state.
• For the use cases where users would want a traditional package-based experience, this should provide it to them with very minimal overhead other than redirecting everything through overlayfs. They can still ad-hoc made modifications to the filesystem, install/remove packages as if the system were provisioned in package mode.
• If we can add the ability to do bootc commit, this would allow users to make adhoc modifications to the system and then commit the system state into a container image. Then users could have system "checkpoints" they could rollback to in the future. (Something like btrfs snapshots would be better at this, but it gives us something similar).
• Since the system is already installed as a bootc system, it means that if the user wanted to transition from the mutable package-based flow to an image-based mode during the system lifecycle, one could do so by disarding the overlay and doing a bootc switch to their desired image without needing to reinstall the system from scratch.

There's a whole lot of technical details being hand-waived away here, but it seems sane to me on the surface. Enough so that I felt it's worth bringing up for wider discussion.

I think this would be a desirable mode for distribution builders to utilize for default system deployment. For something like Fedora Workstation, I think it brings some tangible benefits over the traditional deployment style. The container images can be composed server-side separate from the installation media. Then a "workstation" install becomes using a generic installer image to install a specific container build. If we transition all of the atomic desktops to being bootc-based, then you could easily switch from a "default" workstation to an immutable silverblue or kinoite or whatever, switch back and forth freely, and you could even go back to the "default" experience if you don't like the atomic desktop experience. I think the "free" factory reset capability is a really nice bonus feature. Meanwhile I think users would generally be receptive to such an arrangement because if they choose not to use any of the image-based workflow they can continue to use the system in the same manner as before.

[dustymabe]

However, the difference would be that after the initial installation, the system would have a persistent read-write overlay on top of the root.

With this persistent read-write overlay I'm guessing it makes bootc upgrade useless unless you want to throw away the overlay?

[jeckersb]

With this persistent read-write overlay I'm guessing it makes bootc upgrade useless unless you want to throw away the overlay?

Yeah I don't see any sane way to try and keep track of what has changed and try to do a "rebase" by reusing the overlay content onto a newer container image.

In this mode you'd have two options for upgrades:

1. dnf update just like you would in traditional package mode
2. bootc switch to a new image and discard the overlay

[cgwalters]

I think a downside of this mode is basically that if you don't commit, then you have an ever-growing delta versus the initial installation. Though maybe for most desktops that's not a bad tradeoff, and of course we could make it so that bootc upgrade --pull-only in this world actually just updated the reference base etc.

[SNThrailkill]

If we did try this approach where we swap out the underlying base image and add the overlays ontop like rpm-ostree does today, how can we guarantee that the new staged version will come up? What if the base image does something incompatible that breaks one of the overlays? Im not sure I know of a way to check that before we actually try to boot.

[jeckersb]

If we did try this approach where we swap out the underlying base image and add the overlays ontop like rpm-ostree does today, how can we guarantee that the new staged version will come up? What if the base image does something incompatible that breaks one of the overlays? Im not sure I know of a way to check that before we actually try to boot.

In this proposal each overlay is pinned to the exact base image that it was created from. Switching to a different image always involves creating a new (empty) overlay on the new image. If you decided to rollback to the previous image though, we could save the old overlay and re-use it to go back to the previous (pre-switch) state.

[tulilirockz]

I have a fun thing to take inspiration from: If your systemd build is compiled with systemd-sysext support, you can run sudo systemd-sysext merge --mutable=yes to enable a persistent overlay for /usr and /opt. Not exactly what is being proposed here, but could be used as a reference!

systemd-sysext stores those "layers" into /var/lib/extension.mutable/* and leaves them there, I'd supposed that having something like this on bootc, but instead using ostree commit/composefs hashes as the identifiers and cleaning them up later probably would be an interesting way to implement this.