quadlet: ideas for expansion to builds and images

[dustymabe]

/kind feature

Description

The recent addition of quadlet functionality is great. So great that I went out of my way to grab the upstream bits to test it before it even lands in a release or Fedora Rawhide. I have some ideas for expanding this model:

• Support for .build files.
  • Let's support being able to describe a container build in a declarative way. If I pair a .build with a .container file that runs a container from that build then it would be a really nice combination.
  • Even better would be the ability to inline some Containerfile contents for simple containers.
• Support for .image files.
  • It would be really nice to be able to drop down a .image file that tells the system on boot to import a container image from a location, either local or remote. i.e. a user could pre-load oci-archive files into a location on the host and then drop in .image files that tell podman to load them into container storage on first boot.

I imagine both .build and .image files should be processed before .container file containers are attempted to be started.

These are just ideas. WDYT?

[vrothberg]

@alexlarsson WDYT?

The .image idea can be interesting to pre-fetch images on boot - assuming .image units are started prior to the containers. But, the containers could very well just directly use the various transports in the image section. For instance, image=oci-archive:/tmp/my-archive.tar.gz.

For .build I would avoid reinvent the wheel and embed a Dockerfile/Containerfile. Can you elaborate on a real-world use case? It seems quite exotic to me to build containers at boot time.

[rhatdan]

Well this would match up with docker compose --build use case. Where the users want to provide a unit file and a Containerfile describing an image, but not have to store the image at a registry.

.image could come in handy for systems with image_pull=never being set in containers.conf.

[dustymabe]

@alexlarsson WDYT?

The .image idea can be interesting to pre-fetch images on boot - assuming .image units are started prior to the containers. But, the containers could very well just directly use the various transports in the image section. For instance, image=oci-archive:/tmp/my-archive.tar.gz.

Correct. I just learned about being able to pass oci-archive:/path/to/archive to podman run today. Definitely useful.

Still I think this could be useful in a scenario where you want the image to be update-able. i.e. the container you start uses a remote registry image, but the .image you pre-baked into the image is just a way to seed that in a fast way. i.e. think an amazon ec2 instance with a secondary EBS volume with your organization's most recent builds of their container images in it. When a new instance starts up in populates local quay.io/org/mycontainer and then when the container runs it contacts the registry but realizes it already has the container image and doesn't need to download anything.

For .build I would avoid reinvent the wheel and embed a Dockerfile/Containerfile. Can you elaborate on a real-world use case? It seems quite exotic to me to build containers at boot time.

I guess I'm the exotic one here :) - I really like self contained solutions to problems. For simple problems I really like being able to start a Fedora CoreOS node, have it build any container image it needs and then start the container from that. On system update/reboot (happens about every other week) the container gets built again and container started again from the updated container.

Completely self contained. Doesn't rely on a remote registry or build tooling. Manages itself. One place to go and look for problems.

Sometimes registries and build systems are a blocker. For example, we have quay.io (which is great). It can build your containers for you, etc etc.. Unless you need multi-arch. Then you need to build your containers yourself and push them there and manage registry push secrets and all that jazz.

Or maybe you have secret stuff in your containers. Now you might not want to push them to a registry or you need to push them to a private registry ($$).

I think all I'm trying to illustrate here is that I don't think the use case(s) for this are that far outside of the norm. I don't host any containers in a registry and I have quite a few systems stood up that use containers. Here are examples (the few that are public) that use this model:

• https://pagure.io/releng/archive-repo-manager/blob/0600740dad82eb1115b4fe3c5bc32538045bbc54/f/archive-repo-manager.bu#_90
• https://github.com/coreos/fedora-coreos-pipeline/blob/010e22446c1d81b918022ef97233cdccce2a402a/multi-arch-builders/builder-common.bu#L92

[vrothberg]

Well this would match up with docker compose --build use case. Where the users want to provide a unit file and a Containerfile describing an image, but not have to store the image at a registry.

kube play already supports building images. So Quadlet's .kube files can be extended to support it.

.image could come in handy for systems with image_pull=never being set in containers.conf.

Let's consider this use case. --pull=never was motivated by avoiding image pull during boot as it would slow down things considerably. An .image file would still slow things down for this use case as it would pull down the image at boot time. IMO --pull=never requires the container images to be part of the base OS (prepulled before boot).

I think all I'm trying to illustrate here is that I don't think the use case(s) for this are that far outside of the norm. I don't host any containers in a registry and I have quite a few systems stood up that use containers.

Thanks for taking the time to elaborate, @dustymabe! Did you consider using play kube's build functionality?

I think Quadlet should focus on running containers/pods/kube yaml in systemd. Building containers in systemd feels outside that focus, especially since kube play has such a functionality to compete with compose.

Note that I am not rejecting the idea but I want to force us to consider existing alternatives and to balance pros and cons. Maintaining too many ways of essentially doing the same thing is expensive long term (development, maintenance, testing, etc.).

[alexlarsson]

I feel that quadlet is basically a frontend for "podman run" (and "podman kube play"). I don't think adding a bunch of other stuff to it is a great idea. Code-wise it becomes quite convoluted, as we can't essentially run anything at all during generation, so we would have to generate a set of unit files with some ordering such that the complete thing works. Quite hairy, and will have lots of weird corner cases.

[dustymabe]

I feel that quadlet is basically a frontend for "podman run" (and "podman kube play"). I don't think adding a bunch of other stuff to it is a great idea. Code-wise it becomes quite convoluted, as we can't essentially run anything at all during generation, so we would have to generate a set of unit files with some ordering such that the complete thing works. Quite hairy, and will have lots of weird corner cases.

The ordering problem is easily solved with systemd targets IIUC. Just make sure the container units run After=podman-systemd-images.target podman-systemd-builds.target.

[dustymabe]

Well this would match up with docker compose --build use case. Where the users want to provide a unit file and a Containerfile describing an image, but not have to store the image at a registry.

kube play already supports building images. So Quadlet's .kube files can be extended to support it.

I've looked at kube play. It's just not for me. I feel like if you are not coming from a kubernetes world OR if kubernetes isn't your end goal then it's not really an ideal solution for this use case. The "compose" world is definitely easier to understand.

I also feel like it has a branding problem. I think it's going to be hard to get people to take seriously anything that has the word play in it.

.image could come in handy for systems with image_pull=never being set in containers.conf.

Let's consider this use case. --pull=never was motivated by avoiding image pull during boot as it would slow down things considerably. An .image file would still slow things down for this use case as it would pull down the image at boot time. IMO --pull=never requires the container images to be part of the base OS (prepulled before boot).

Not necessarily. In the example I gave in #16775 (comment) (EC2 instance with sidecar EBS volume) the image files are actually available on a "local" filesystem (secondary disk), they just need to be imported. It would be nice if there was a standardized way to do the import. Of course each user can write their own, but I think we can bring value to everyone by not requiring people to do that.

I think all I'm trying to illustrate here is that I don't think the use case(s) for this are that far outside of the norm. I don't host any containers in a registry and I have quite a few systems stood up that use containers.

Thanks for taking the time to elaborate, @dustymabe! Did you consider using play kube's build functionality?

Yeah. Unfortunately, I don't think it's for me, but that's OK because what I currently have (which is a systemd unit I define myself to do the build on each boot) works well enough.

I think Quadlet should focus on running containers/pods/kube yaml in systemd. Building containers in systemd feels outside that focus, especially since kube play has such a functionality to compete with compose.

Note that I am not rejecting the idea but I want to force us to consider existing alternatives and to balance pros and cons. Maintaining too many ways of essentially doing the same thing is expensive long term (development, maintenance, testing, etc.).

Thanks for the discussion! It's useful to have.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I am converting this to a discussion,since that is where we are with it now.

[Romain-Geissler-1A]

Hi @vrothberg,

Actually today I was looking for this possibility to have quadlet also build containers on the fly, rather than specifying an image ;) And like as @dustymabe I don't feel like kube play is super user friendly for what I want to do.

One way to avoid any problem on your side (ie avoid the hassle of the whole building support, which has many use case) might be to allow providing some kind ImageGenerationCommand instead which would need to output (maybe in a file passed as argument ?) the image name to use, and that command may be any script the user want to use. This script could pull images from somewhere, or build an image with the plethora of build options buildah has.

[dustymabe]

Concerns would be the amount of time it would take to build an image. If I have foobar.container relying on foobar.build but foobar.build takes 5 minutes to complete, what does systemd do?

Systemd is good at ordering things if you tell it to. i.e. use After= like is done here.

[rhatdan]

Yes I know that, I am just wondering about timeouts. If you say After=foobar and foobar takes a huge amount of time, then does systemd cause a timeout by default? If yes, then users might be surprised by this.

[dustymabe]

Correct. There should be some mechanism for the user to specify how long it should take for the the build to happen. For example, we're using TimeoutStartSec=10m here

[rhatdan]

Well I am fine with adding .build, @vrothberg has reservations. But I don't think we would block community from adding support for it.

[vrothberg]

Yes, I do have reservations until we have a design written down. Let's not design in a PR but have something sketched out. After that, the implementation is just a detail :^)

[paomian]

anything update ？I see support for .image, it seems .build isn't in the documentation yet.

[ygalblum]

If they have both, then a systemd unit file will be generated for both commands. But, they will be started only if the user explicitly starts them or starts another service that depends on them. So, assuming the user starts the service generated for the .container file, the service that corresponds with the value of the Image key will be started

[vrothberg]

maybe @vrothberg can help answer since I think he was #17043 (reply in thread).

Thanks! Yes, starting with a design first sounds good. But I also encourage somebody to volunteer to maintain the desired .build functionality. I will effectively boil down to chasing/copying the CLI flags of podman build (see podman build --help) along with adding tests etc. It's a lot of (continuous) work that somebody should sign up for.

[jmaibaum]

Well, I am in the process of implementing .build functionality in the quadlet-build branch of my fork. It comes along nicely and I would assume that by now it's mostly adding more (all?) remaining comman line args along with doc+tests, before I can open a PR.

As noted above, I am both low on free time, and also first-time Go user (though it seems fairly straight-forward so far), so making slow progress only. Cannot yet say if I have enough time to maintain this indefinitely should it get merged (at least I can likely not commit to any "I will promptly address everything that comes up" expectations).

Yet, I am happy to discuss if the approach is correct and if I should continue like this or not. Feel free to address any concerns.

[jmaibaum]

My PR is in: #22694

Happy to hear what you think :)

[jmaibaum]

Thanks again for merging. Looking forward to see .build support in a released version! :)

[lgranie]

You can manage to build your personal image from a directory containing a Containerfile by adding a [Service] section with a WorkingDirectory directive in the .kube file.
For exemple :
[Kube]
Yaml=/home/podman-runner/.config/containers/pods/my-pod.kube

[Service]
WorkingDirectory=/home/podman-runner/.config/containers/pods/

|----- pods
|----- my-pod.kube
|----- my-image
|        |---- Containerfile

[ygalblum]

@lgranie while your solution is perfectly fine, please note that Quadlet has a key (SetWorkingDirectory) for this.

So, the following:

[Kube]
Yaml=/home/podman-runner/.config/containers/pods/my-pod.kube
SetWorkingDirectory=yaml

Will yield the exact same results without having to write the path twice.

[gnat]

For .build I would avoid reinvent the wheel and embed a Dockerfile/Containerfile. Can you elaborate on a real-world use case? It seems quite exotic to me to build containers at boot time.

App containers where you don't want to set up / host an entire container registry just for your app image.

Would be awesome to embed- cutting down on dependency hell.

I think once we get that, many people can simply migrate away from Docker Compose / Podman Compose, as quadlet / systemd could do the orchestration itself.

[ygalblum]

Closing this discussion following the merge of #22694