Support persisting changes to a stage mount

[jlebon]

We'd like to be able to do something like:

FROM quay.io/fedora/fedora-minimal AS final

FROM quay.io/fedora/fedora-minimal AS build
RUN --mount=type=bind,from=build,target=/final,persist echo foobar > /final/foobar.txt

FROM final

Where changes to the stage done through its mount are persisted (i.e. committed) rather than lost as part of the throwaway overlay. (Notice the persist mount option there.)

The use case for this is when you want to use an image with some kind of build tool to operate on the final image without wanting to ship that build tool itself in the image. The buildah answer to this currently are the buildah from ... and buildah commit primitives. The proposal here is to support a Containerfile-native way to do this, which would have major UX and CI advantages.

That said, two concrete use cases to motivate this further:

1. In Project Hummingbird, we ship distroless images (i.e. without dnf). Multi-stage builds is fine for the initial base image build. But we also want users to be able to extend our images using dnf without actually needing dnf in the image. So e.g. it would permit something like:

   FROM quay.io/hummingbird/rust:latest-builder AS builder
   COPY . /build
   WORKDIR /build
   RUN dnf install -y libfoobar-devel && cargo build --release
   
   FROM quay.io/hummingbird/core-runtime AS final
   COPY --from=builder /build/target/release/myapp /usr/local/bin/myapp
   
   # add runtime dep libfoobar to final image using dnf from builder image
   FROM quay.io/hummingbird/rust:latest-builder
   RUN --mount=type=bind,from=final,target=/final,persist dnf install -y --installroot /final ... libfoobar
   
   FROM final

   (This could be shorter, but keeping it similar to the traditional multi-stage flow for clarity.)

2. In bootable containers, users want to be able to run their Ansible playbooks to configure their image at build time. See e.g. https://github.com/nzwulfin/bootc-django (that uses the host Ansible) and https://gitlab.com/fedora/bootc/examples/-/blob/main/ansible-firewalld/Containerfile?ref_type=heads (installs Ansible, runs it, then uninstalls it). This would become:

   FROM quay.io/centos-bootc/centos-bootc:stream10 AS final
   
   # NB: there might be a well-known user-facing ansible image that exists already
   FROM quay.io/centos/centos:stream10 AS ansible
   RUN dnf install -y ansible-core
   COPY . /src
   RUN --mount=type=bind,from=final,target=/final,persist ansible-playbook -c chroot -i /src/inventory ...
   
   FROM final

[nalind]

I think that this would make parallelizing stages, when one uses the contents of a stage that is updated this way by another, quite a bit more complicated, since a lot of that logic currently depends on being able to "know" when a stage has been completed. I think you're already familiar with how complicated that area of the implementation is.

I'm also unclear on what a RUN instruction with a bind mount that doesn't specify this "persist" option would expect to see.

That aside, I do agree it'd be pretty useful to be able to have other tools muck around with the rootfs of a stage that's being built while it's being built (stress the "while it's being built"), and think we might be able to get that much by leaning on existing features.

The minimal case is mounting an image with RUN --mount=type=bind and running static binaries it provides, letting them do their thing right there.

Things get messier as soon as that image uses shared libraries and includes configuration files, like the case above where you'd want to run dnf, and have it modify the rootfs for the current stage. dnf can be told to operate on a chrooted directory, and, with some finagling, you can actually bind mount the rootfs of the stage into a subdirectory under the mountpoint where the image that includes dnf is mounted, and then run dnf under chroot, pointing it the bind mount as a chroot to operate on.

I started roughing that out. You have to create a new user namespace in order to be able to create bind mounts since we don't provide cap_sys_admin by default, and then you have to create a new mount namespace owned by that user namespace, and you want to not set up any new ID mappings in the nested user namespace, so you need cap_setfcap, but that's provided by default, along with cap_sys_chroot.

Tools like dnf also tend to want to reach out over the network, so you tend to want to bind-mount the resolver configuration. That ends up producing a lot of boilerplate, but I think the approach could be of general use, so I threw together https://github.com/nalind/chrooty to see if it could be made a little more streamlined. It's not as flexible as what you've described, and I'm sure I've overlooked some things, but maybe it's useful enough to solve the cases you're seeing.

[jlebon]

Heh, re. chrooty I basically had the same idea before filing this issue and that would work yes. Another approach I tried was literally to run buildah nested as part of the build (e.g. FROM quay.io/buildah/stable, then buildah from ..., buildah mount ..., do operations, then buildah push to an OCI archive, and then use the FROM oci-archive: trick). Though both those approaches require privileges, and have an awkward UX.

At a high-level, it seems quite unfortunate that we're re-implementing containerization when running in a tool which natively supports this. The question I'm trying to answer is more: how could we achieve a nicer UX for this use case if we could extend buildah in a way that combines pieces it already knows how to do?

I think that this would make parallelizing stages, when one uses the contents of a stage that is updated this way by another, quite a bit more complicated, since a lot of that logic currently depends on being able to "know" when a stage has been completed.

Yeah, I had a look at that. It's obviously doable but somewhat gnarly.

Hmm, and what if instead we modeled this at the FROM level, e.g.

FROM --customize=with=mytoolsimg,rootfs=/target quay.io/hummingbird/core-runtime AS final
# RUN instructions here actually run in a mytoolsimg container with quay.io/hummingbird/core-runtime
# mounted at /target. The resulting `final` image is committed from `/target`.

?

So this essentially flips things. The stage's rootfs is the one with the transient overlay, and changes to the /target rootfs are what get committed at the end as a new image. What's nice about this is that (1) it more clearly marks the whole stage as a "outer tool" building stage, and (2) it retains the AS foo syntax to refer to the customized image as usual like any other stage.

(But to be clear, open to other ideas/syntaxes to express this!)