Pushing and pulling arbitrary data as OCI artefacts (on macOS and Linux)

[ericcurtin]

Objective:

Develop a tool to create container artefacts from scratch without using a container runtime. So we can push/pull from quay.io

Key Requirements:

Tool should run natively on macOS/Linux.
The ability to store and handle large files (hundreds of GB).
Eliminate the need for intermediate tarball copies to save disk space. (nice to have, stream/create the artefact on the fly to avoid duplicating data on disk, this is probably a long-term goal, not for initial implementation)
Doesn't necessarily have to be part of podman (for example skopeo does pushing/pulling)

Some initial thoughts:

Create a tarball as the layer, digest it to get a diffID, and populate the config and manifest blobs with these details.

ocidir-rs and skopeo could be part of the solution.

Tagging @rhatdan @cgwalters @mtrmac @nalind

[ericcurtin]

We want to implement a solution like this (we had the same idea, which is normal, consolidating on the same ideas, realized LocalAI does it already on Linux):

mudler/LocalAI#2628

Linux support is just "podman build" but would like a macOS compatible solution.

bootc has similar requirements. And this could be useful for any type of arbitrary files data that wants to use OCI registries.

[ericcurtin]

A generic CLI that uses this:

https://github.com/containers/ocidir-rs/

could be very useful.

One use case is this thing:

https://github.com/containers/ramalama

That only supports pulling from Ollama registry right now.

[ericcurtin]

The imaginary feature:

podman --runtime noruntime build

is sort of the main crux right now (for macOS). At least for https://github.com/containers/ramalama

[cgwalters]

Do you really want to store it as a container image? Wouldn't this be better as an OCI artifact? In which case, oras already meets the requirements I think.

[ericcurtin]

I got feedback from the package maintainer of that package... Said it was difficult to keep stable, maybe those woes are gone... It's also tempting to use FROM scratch images to stay compatible with LocalAI, but we don't have to mudler/LocalAI#2628

[mtrmac]

First answer the image/artifact question, please; implementation details later.

Create a tarball as the layer

Is that a requirement or an implementation suggestion?

[ericcurtin]

Just something I copy pasted from previous chats, from my perspective it's neither, above was just a best effort to condense all previous conversations and summarise here.

[rhatdan]

One key feature needed for this tool if it will not use skopeo for pulling/pushing, would be to use containers/image. We need to make sure that mirror systems continue to work with the same configuration as podman, buildah and skopeo support.

I like the idea of just using artifacts rather then container images with from scratch; copy foo /bar. Especially if this eliminates having to create a tar ball from and existing, potentially huge, file.

[cgwalters]

One key feature needed for this tool if it will not use skopeo for pulling/pushing, would be to use containers/image.

Yes, oras not using c/image has a notable ergonomic hit.

Actually...I am honestly not in the loop enough as to the state of OCI artifacts in podman/buildah/skopeo...digging around a bit I found #21785

And we debated this a lot in containers/buildah#5091 but...I don't know where that all ended up.

[rhatdan]

We ended up with no conclusion.

[arewm]

I have used buildah source and buildah manifest add <artifact> a little bit to create OCI artifacts but others prefer the interface of oras (partially because oras had fewer limitations and because oras supports the referrer's api). For pulling the artifacts, oras has become the default that many use.

[cgwalters]

If we're debating artifacts my proposal is to close this discussion and go back to using containers/buildah#5091 as there are too many threads and that one I thought was most clear/technical/actionable.

[cgwalters]

(I went ahead and closed but, if someone disagrees strongly, feel free to reopen)