Make Podman support more prominent in the documentation

[rgaiacs]

By

• using "container" instead of Docker when possible
• mention Podman configuration

[rgaiacs]

Please correct me if I got something wrong.

Before #1402, repo2docker used Docker SDK for Python for all operations with Docker. The Docker client was created using from_env() that read information from the DOCKER_HOST environment variable. This allowed users to configure DOCKER_HOST to point to their Podman installation, see procedures.

After #1402, repo2docker did not work with Podman. Changes in DockerEngine included in this pull request, restore the support to Podman.

Regarding container technology, the big players are

• Docker
• Podman (backed by Red Hat)
• Singularity (from Sylabs)
• Apptainer (fork of Singularity and under the umbrella of the Linux Foundation)

Podman keeps some level of compatibility with Docker. As far as I know, Singularity / Apptainer are incompatible with Docker.

My first question is, should repo2docker also support Podman as first class container engine? I mean: repo2docker documents how to use Podman and run tests with Podman.

If the answer is "yes", a follow up question is what would be the road map for repo2podman, a plugin for repo2docker that @manics developed.

[minrk]

My understanding was that podman was supported as long as it the engine backed by the docker cli (e.g. I use colima, not docker engine), but podman cli is not supported first-class in this package. Is that accurate, @manics?

I agree generally that pointing most language to OCI terms is good, but I don't know what commitment we are making in terms of cli compatibility assumptions.

[manics]

Originally repo2docker should've worked fine with Podman when run as a service since podman has a docker compatible API:

podman system service --time 0
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

After #1402 we're now using the Docker CLI. In general the Podman CLI should be compatible, but I haven't looked at how good buildx support is. If it currently works I've no objection to supporting if, and ensuring that any future changes to repo2docker only use buildx features that podman supports.

[minrk]

If we've got tests running that verify it (to a reasonable degree), I'm happy to make the support official.

[rgaiacs]

After

• change the code to do base the logic in DOCKER_HOST
• add container_cli as a GitHub Actions matrix dimension

most of the tests using Podman fails. One of the tests fails in

repo2docker/repo2docker/app.py

Lines 633 to 634 in 358888c

	image = client.inspect_image(self.output_image_spec)
	image_workdir = image.config["WorkingDir"]

When I try to investigate it locally, I observe

>>> import docker
>>> client = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
>>> image = client.inspect_image("localhost/r2d-tests-dockerfile-2fsimple-1759752459")
>>> image.config
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    import platform
    ^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'config'
>>> dir(image)
['__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__ior__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__ror__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'copy', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values']
>>> image.keys()
dict_keys(['Id', 'RepoTags', 'RepoDigests', 'Parent', 'Comment', 'Created', 'ContainerConfig', 'DockerVersion', 'Author', 'Config', 'Architecture', 'Os', 'Size', 'VirtualSize', 'GraphDriver', 'RootFS', 'Metadata', 'Container'])
>>> image["Config"]
{'Hostname': '', 'Domainname': '', 'User': '1000', 'AttachStdin': False, 'AttachStdout': False, 'AttachStderr': False, 'Tty': False, 'OpenStdin': False, 'StdinOnce': False, 'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LANG=C.UTF-8', 'GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D', 'PYTHON_VERSION=3.10.18', 'PYTHON_SHA256=ae665bc678abd9ab6a6e1573d2481625a53719bc517e9a634ed2b9fefae3817f', 'HOME=/tmp'], 'Cmd': ['/bin/sh', '-c', '"/bin/sh"'], 'Image': '', 'Volumes': None, 'WorkingDir': '', 'Entrypoint': None, 'OnBuild': None, 'Labels': {'io.buildah.version': '1.40.0', 'repo2docker.ref': 'None', 'repo2docker.repo': 'local', 'repo2docker.version': '2025.08.0+61.gced04a5'}}
>>> image["Config"]["WorkingDir"]
''

@manics do you know what might be happening?

[rgaiacs]

In 48e9243, tests with Podman were failing because Podman's service was not activate. This was fixed in 4e7ed9e and documented in ef36a9c.

Test of Dockerfile configuration with Podman continues to fail in

repo2docker/repo2docker/app.py

Lines 632 to 634 in 358888c

	if self.volumes:
	image = client.inspect_image(self.output_image_spec)
	image_workdir = image.config["WorkingDir"]

I can reproduce the error locally with

import docker

a = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
i = a.inspect_image("docker.io/library/httpd:latest")
i.config

that returns

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    import platform
    ^^^^^^^^
AttributeError: 'dict' object has no attribute 'config'

but I can access the information using

i["Config"]["WorkingDir"]

I tested the

import docker
a = docker.APIClient(base_url="unix:///run/user/1000/podman/podman.sock")
i = a.inspect_image("docker.io/library/httpd:latest")
i["Config"]["WorkingDir"]

with Docker and it works.

[rgaiacs]

A few tests are failing.

WorkingDir is missing

• tests/dockerfile/binder-dir/verify::check-tmp
• tests/dockerfile/simple/verify::check-tmp

in test (24.04, 3.13, podman, dockerfile).

I wrongly assume that in

repo2docker/repo2docker/app.py

Lines 632 to 634 in 358888c

	if self.volumes:
	image = client.inspect_image(self.output_image_spec)
	image_workdir = image.config["WorkingDir"]

the client was APIClient when, in fact, the client is DockerEngine.

DockerEngine's inspect_image looks OK but there are cases where docker image inspect and podman image inspect produce different outputs.

I will try to investigate this upstream. I asked containers/podman#27313.

buildx support

• tests/unit/test_app.py::test_extra_buildx_build_args
• tests/unit/test_connect_url.py::test_connect_url
• tests/unit/test_editable.py::test_editable
• tests/unit/test_editable.py::test_editable_by_host
• tests/unit/test_users.py::test_user
• tests/unit/test_volumes.py::test_volume_abspath
• tests/unit/test_volumes.py::test_volume_relpath

in test (24.04, 3.13, podman, unit).

My suspicious is that the test fails because of some "obscure" difference between Podman and Docker.

[rgaiacs]

With commit efe313d256ffba5626a26c748a8ce3774ad3c5b7, when I run

pytest -s tests/unit/test_editable.py::test_editable

the log shows

[Warning] one or more build args were not consumed: [NB_USER]

Why does the above warning happened? Is --user-id not supported for Dockerfile configuration?

[manics]

DockerEngine's inspect_image looks OK but there are cases where docker image inspect and podman image inspect produce different outputs.

I repo2podman I found passing the format arg --format '{{json .}}' worked:
https://github.com/manics/repo2podman/blob/b772bb818dbdc8f513a01bbb0c02eed4848d371d/repo2podman/podman.py#L259-L264

[rgaiacs]

This is ready for a review and I would suggest to avoid increase it more.

The tests now run and pass for Podman as for Docker. Because Podman is, by default, rootless, I configured tests related to bind mount to be skipped, read #1483 for details. I also skipped the tests related with the registry as they have some bind mount, see

repo2docker/tests/norun/test_registry.py

Lines 42 to 43 in 443b851

	"--mount",
	f"type=bind,src={cert_dir},dst=/opt/certs",

The GitHub Action runner ubuntu-24.04 includes Podman 4.9.3 that is a good number of versions older than the latest Podman release (5.6.2). In another pull request, we could have miniconda installing podman from conda-forge.

[rgaiacs]

@minrk wrote

If we've got tests running that verify it (to a reasonable degree), I'm happy to make the support official.

Excepted for some tests mentioned in #1471 (comment), we have all existing tests passing for Podman and Docker.

@manics wrote

I don't think we need to run all tests with Podman, just a subset

and I agree that we could avoid run all the tests a second time for Podman. Before I amend this pull request, do you have a strong opinion of which or how many tests should be run with Podman?

Instead of using @manics suggestion to select the subset of tests for Podman with include:, I prefer to go down one level and use pytest.mark to annotate the tests that we want to run with Podman.

[manics]

How about:

• The UI test https://github.com/jupyterhub/repo2docker/tree/main/tests/ui
• Unit tests since they should be quick https://github.com/jupyterhub/repo2docker/tree/main/tests/unit
• one of https://github.com/jupyterhub/repo2docker/tree/main/tests/dockerfile
• https://github.com/jupyterhub/repo2docker/tree/main/tests/conda/py-r-postbuild-file since it involves R, Python and Postbuild
• https://github.com/jupyterhub/repo2docker/tree/main/tests/pipfile/setup-py-explicit-in-binder-dir
  ?

If you're using pytest.mark perhaps mark them as something like "quick/quicktest/fast/etc" instead of "podman" since it's more generally useful if you don't want to run the full test suite?

[rgaiacs]

The tests with Podman were stuck waiting for a server with the label ubuntu-.

7f6b65e resolved the problem.

[manics]

Strangely the failing test is runingn out of disk space
INFO repo2docker:app.py:824 Error: committing container for step {Env:[NB_USER=runner NB_UID=1001 REPO_DIR=/srv/repo PATH=/home/runner/.local/bin:/srv/repo/.local/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DEBIAN_FRONTEND=noninteractive LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 SHELL=/bin/bash USER=runner HOME=/home/runner APP_BASE=/srv CONDA_DIR=/srv/conda NB_PYTHON_PREFIX=/srv/conda/envs/notebook NPM_DIR=/srv/npm NPM_CONFIG_GLOBALCONFIG=/srv/npm/npmrc NB_ENVIRONMENT_FILE=/tmp/env/environment.lock MAMBA_ROOT_PREFIX=/srv/conda MAMBA_EXE=/srv/conda/bin/mamba CONDA_PLATFORM=linux-64 KERNEL_PYTHON_PREFIX=/srv/conda/envs/notebook CONDA_DEFAULT_ENV=/srv/conda/envs/notebook] Command:run Args:[TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '] Flags:[] Attrs:map[] Message:RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} ' Original:RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '}: copying layers and metadata for container "c60d8adc11015838b3f6e819afdad9f6054ce954adf3118bb76ca8441f83c073": writing blob: adding layer with blob "sha256:c4b8ce47904a0e175bde526208956231b1dfb32611a0a98552d58c3b96036732": processing tar file(write /srv/conda/envs/notebook/libexec/gcc/x86_64-conda-linux-gnu/9.5.0/cc1obj: no space left on device): exit status 1

[rgaiacs]

Strangely the failing test is runingn out of disk space

Is a suitable solution to call container.remove() and image.remove()? Maybe after

repo2docker/tests/conftest.py

Line 92 in ca21e1b

app.wait_for_container(container)

[manics]

There's certainly workarounds we can add, like removing images/containers, or freeing up space on the VM https://github.com/manics/action-free-disk-space
but I'm surprised it's a problem in the first place.

[rgaiacs]

I'm surprised it's a problem in the first place.

I'm also surprising as the same test for Docker does not have the storage limit problem.

[rgaiacs]

6baaf08 failed when running the tests with the same "no space left on device" message. I don't know why the step to remove the old image did not avoid this problem.

In 7f6b65e, the only tests to fail were

• tests/conda/r3.6-target-repo-dir-flag/verify::build
• tests/conda/r3.6-target-repo-dir-flag/verify::verify
• tests/conda/r3.6-target-repo-dir-flag/verify::check-tm

but, now, there list of tests to fail increased to include

• tests/conda/py36-postBuild/verify::build
• tests/conda/py36-postBuild/verify::verify
• tests/conda/py36-postBuild/verify::check-tmp

In containers/podman#16192, the user rhatdan wrote

There have been some improvements in Buildah since podman 4.1, which might help with this.

@minrk and @manics what do you think of also disable the test of conda for podman until

1. Ubuntu 26.04 LTS be released by Canonical
2. GitHub update the runners to include a runner with Ubuntu 26.04 LTS

given that Ubuntu 24.04 LTS used for test has a older version of Podman (4.9).

[rgaiacs]

@manics and @minrk I disabled the test of conda for podman as mentioned in the previous message. Now, all the tests pass. This includes the base, dockerfile and unit test with podman.

I would like to merge this on Monday, 19 January as this pull request is already big enough and was reviewed multiple times.