Skip to content

Commit 8ff237f

Browse files
authored
Merge pull request #1471 from rgaiacs/document-podman-support
Make Podman support more prominent in the documentation
2 parents ca21e1b + 9032292 commit 8ff237f

File tree

11 files changed

+194
-46
lines changed

11 files changed

+194
-46
lines changed

.github/workflows/test.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ jobs:
5353
matrix:
5454
ubuntu_version: ["24.04"]
5555
python_version: ["3.13"]
56+
container_cli:
57+
- docker
5658
repo_type:
5759
- base
5860
- conda
@@ -68,11 +70,34 @@ jobs:
6870
- norun
6971
# Playwright test
7072
- ui
73+
# If the item to be include would overwrite a value,
74+
# it must have all the fields!
7175
include:
7276
# The earliest actions/setup-python versions depend on the runner.
7377
- ubuntu_version: "24.04"
7478
python_version: "3.10"
79+
container_cli: docker
7580
repo_type: venv
81+
# Run all tests for both Docker and Podman is expensive.
82+
# We only run the "smoke" test for Podman.
83+
- ubuntu_version: "24.04"
84+
python_version: "3.13"
85+
container_cli: podman
86+
repo_type: base
87+
# We are observing "no space left on device" on GitHub Actions,
88+
# https://github.com/jupyterhub/repo2docker/pull/1471#issuecomment-3744990787
89+
# - ubuntu_version: "24.04"
90+
# python_version: "3.13"
91+
# container_cli: podman
92+
# repo_type: conda
93+
- ubuntu_version: "24.04"
94+
python_version: "3.13"
95+
container_cli: podman
96+
repo_type: dockerfile
97+
- ubuntu_version: "24.04"
98+
python_version: "3.13"
99+
container_cli: podman
100+
repo_type: unit
76101

77102
steps:
78103
- uses: actions/checkout@v6
@@ -104,6 +129,11 @@ jobs:
104129
105130
- name: Run pytest
106131
run: |
132+
if [ "${{ matrix.container_cli }}" = "podman" ]
133+
then
134+
systemctl --user start podman.socket
135+
export DOCKER_HOST=unix://$(podman info --format '{{.Host.RemoteSocket.Path}}')
136+
fi
107137
pytest --verbose --color=yes --durations=10 --cov=repo2docker tests/${{ matrix.repo_type }}
108138
109139
- uses: codecov/codecov-action@v5

docs/source/index.md

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
11
# Welcome to `repo2docker`'s documentation
22

3+
```{important}
4+
Despite the name, `repo2docker` can be used by container technology other than [Docker](https://docs.docker.com/engine/), for example [Podman](https://podman.io/).
5+
```
6+
37
`repo2docker` lets you **reproducibly build and run user environment container images for interactive computing and data workflows from source code repositories**. Optionally, the container image can be pushed to a Docker registry.
48

59
Also, `repo2docker` is the tool used to build container images for [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) and the tool used by [BinderHub](https://binderhub.readthedocs.io) to build images on demand.
610

711
::::{grid}
812
:::{grid-item-card} 🔧 Build reproducible data science environments from repositories
9-
Build a reproducible data science environment as a Docker image and execute code interactively. Use many [configuration files](#config-files) to control language, tools, and setup instructions.
13+
Build a reproducible data science environment as a container image and execute code interactively. Use many [configuration files](#config-files) to control language, tools, and setup instructions.
1014
:::
1115
:::{grid-item-card} 🚀 Deploy environments in JupyterHub or Binder
12-
Push environment images to a Docker registry for re-use in data science environment services like [JupyterHub](https://jupyterhub.readthedocs.io) or [a Binder instance](https://mybinder.org), or for other communities to build upon your base environment.
16+
Push environment images to a container registry for re-use in data science environment services like [JupyterHub](https://jupyterhub.readthedocs.io) or [a Binder instance](https://mybinder.org), or for other communities to build upon your base environment.
1317
:::
1418
:::{grid-item-card} ☁️ Host repositories in many providers
1519
Host repositories in: a Git server like [GitHub](https://github.com/) or [GitLab](https://gitlab.com/), an open science repository like [Zenodo](https://zenodo.org) or [Figshare](https://figshare.com), a hosted data platform like a [Dataverse installation](https://dataverse.org/), an archive like the
@@ -19,7 +23,7 @@ Host repositories in: a Git server like [GitHub](https://github.com/) or [GitLab
1923

2024
## What is a user environment container image and why would I build one with `repo2docker`?
2125

22-
A **user environment container image** contains the entire software environment that a user may access from an interactive data science session. For example, it might contain many **programming languages**, **software for data analysis**, or even **content files and datasets** available to anybody that accesses that environment. Container images are built with [Docker](https://www.docker.com/), a standard open source tool for defining, building, and deploying images.
26+
A **user environment container image** contains the entire software environment that a user may access from an interactive data science session. For example, it might contain many **programming languages**, **software for data analysis**, or even **content files and datasets** available to anybody that accesses that environment. Container images are built in accordance with the spectifications published by the [Open Container Initiative](https://opencontainers.org/).
2327

2428
Many data science platforms and services like [JupyterHub](https://jupyterhub.readthedocs.io) and [Binder](https://mybinder.org) launch interactive data science sessions **with a user environment container image attached**, meaning that the user gains access to whatever is in the container image. In short, this allows somebody to define and build the user image one time, in a way that users can reproducibly re-use many times.
2529

@@ -43,9 +47,9 @@ repo2docker <source-repository>
4347
It performs these steps:
4448

4549
1. Inspects the repository for [configuration files](#config-files). These will be used to build the environment needed to run the repository.
46-
2. Builds a Docker image with an environment specified in these [configuration files](#config-files).
50+
2. Builds a container image with an environment specified in these [configuration files](#config-files).
4751
3. Runs the image to let you explore the repository interactively via Jupyter notebooks, RStudio, or many other interfaces (this is optional).
48-
4. Pushes the images to a Docker registry so that it may be accessed remotely (this is optional).
52+
4. Pushes the images to a container registry so that it may be accessed remotely (this is optional).
4953

5054
[swhid]: https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html
5155

@@ -55,7 +59,7 @@ Please report [bugs](https://github.com/jupyterhub/repo2docker/issues),
5559

5660
## Get started with `repo2docker`
5761

58-
This tutorial walks you through setting up `repo2docker`, building your first environment image, and running it locally with Docker.
62+
This tutorial walks you through setting up `repo2docker`, building your first environment image, and running it locally with a container engine.
5963

6064
```{toctree}
6165
:maxdepth: 2

docs/source/start.md

Lines changed: 48 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,47 +2,72 @@
22

33
This tutorial guides you through installing `repo2docker` and building your first environment image.
44

5-
(install)=
5+
## Prerequisite
66

7-
## Install `repo2docker`
7+
### Python
88

9-
`repo2docker` requires Python 3.6 or above on Linux and macOS.
9+
`repo2docker` requires Python 3.6 or above.
1010

11-
:::{admonition} Windows support is experimental
11+
### Container Engine
1212

13-
This [article about using Windows and the WSL](https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly) (Windows Subsystem for Linux or
14-
Bash on Windows) provides additional information about Windows and Docker.
15-
:::
13+
`repo2docker` requires a container engine compatible with the specification published by the [Open Container Initiative](https://opencontainers.org/).
1614

17-
### Prerequisite: Install Docker
15+
#### Docker
1816

19-
Install [Docker](https://www.docker.com), as it is required to build Docker images.
20-
The [Community Edition](https://docs.docker.com/install/) is available for free.
17+
```{important}
18+
Only the [Docker Engine](https://docs.docker.com/engine/) is open source. [Docker Desktop](https://docs.docker.com/get-started/get-docker/) requires a license.
19+
```
2120

22-
Recent versions of Docker are recommended.
21+
Follow [Docker's official installation steps](https://docs.docker.com/get-started/get-docker/).
2322

24-
### Install `repo2docker` with `pip`
23+
#### Podman
2524

26-
```{warning}
27-
The name of the package on [PyPI](https://pypi.org/) is [`jupyter-repo2docker`](https://pypi.org/project/jupyter-repo2docker/) instead of `repo2docker`.
28-
```
25+
Follow [Podman's official installation steps](https://podman.io/docs/installation).
2926

30-
We recommend installing `repo2docker` with the `pip` tool:
27+
After completing the installation of Podman,
3128

32-
```
29+
1. create a [listening service for Podman](https://docs.podman.io/en/latest/markdown/podman-system-service.1.html) by running
30+
31+
```bash
32+
systemctl --user start podman.socket
33+
```
34+
35+
1. configure the `DOCKER_HOST` environment variable following [Podman's official procedure](https://podman-desktop.io/docs/migrating-from-docker/using-the-docker_host-environment-variable#procedure). You might want to configure the `DOCKER_HOST` environment variable to persist in your `~/.bashrc`.
36+
37+
(install)=
38+
39+
## Install `repo2docker`
40+
41+
### Install `repo2docker` with `pip`
42+
43+
It is recommend to install `repo2docker` with the `pip` tool:
44+
45+
```bash
3346
python3 -m pip install jupyter-repo2docker
3447
```
3548

3649
(usage)=
3750

3851
## Build a repository with `repo2docker`
3952

40-
Now that you've installed Docker and `repo2docker`, we can build a repository.
41-
To do so, follow these steps.
53+
Now that you've installed a container engine and `repo2docker`, you can build a repository.
54+
To do so, continue following this guide.
55+
56+
### Start the container engine
4257

43-
### Start Docker
58+
Ensure that the container engine is running.
4459

45-
Follow the [instructions for starting Docker](https://docs.docker.com/engine/daemon/start/) to start a Docker process.
60+
#### Docker
61+
62+
Follow the [offcial instructions for starting Docker](https://docs.docker.com/engine/daemon/start/).
63+
64+
#### Podman
65+
66+
Run
67+
68+
```bash
69+
podman info
70+
```
4671

4772
### Build an image from a URL
4873

@@ -55,12 +80,12 @@ jupyter-repo2docker https://github.com/binder-examples/requirements
5580
You'll see `repo2docker` take the following actions:
5681

5782
1. Inspect the repository for [configuration files](#config-files). It will detect the `requirements.txt` file in the repository.
58-
2. Build a Docker image using the configuration files. In this case, the `requirements.txt` file will correspond to a Python environment.
83+
2. Build a container image using the configuration files. In this case, the `requirements.txt` file will correspond to a Python environment.
5984
3. Run the image to let you explore the repository interactively.
6085

6186
Click the link provided and you'll be taken to an interactive Jupyter Notebook interface where you can run commands interactively inside the environment.
6287

6388
## Learn more
6489

6590
This is a simple example building an environment image for your repository.
66-
To learn more about the kinds of source repositories, environments, and use-cases that repo2docker supports, see [the `repo2docker` user guide](./use/index.md).
91+
To learn more about the kinds of source repositories, environments, and use-cases that `repo2docker` supports, see [the `repo2docker` user guide](./use/index.md).

repo2docker/app.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -589,7 +589,11 @@ def start_container(self):
589589

590590
docker_host = os.environ.get("DOCKER_HOST")
591591
if docker_host:
592-
host_name = urlparse(docker_host).hostname
592+
docker_host_parsed = urlparse(docker_host)
593+
if docker_host_parsed.scheme == "unix":
594+
host_name = "127.0.0.1"
595+
else:
596+
host_name = docker_host_parsed.hostname
593597
else:
594598
host_name = "127.0.0.1"
595599
self.hostname = host_name
@@ -621,7 +625,7 @@ def start_container(self):
621625
"notebook",
622626
"--ip=0.0.0.0",
623627
f"--port={container_port}",
624-
f"--ServerApp.custom_display_url=http://{host_name}:{host_port}",
628+
f"--ServerApp.custom_display_url=http://{self.hostname}:{self.port}",
625629
"--ServerApp.default_url=/lab",
626630
]
627631
else:

repo2docker/docker.py

Lines changed: 64 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,28 @@
1313
from pathlib import Path
1414

1515
from iso8601 import parse_date
16-
from traitlets import Dict, List, Unicode
16+
from traitlets import Dict, List, Unicode, default
1717

1818
import docker
1919

2020
from .engine import Container, ContainerEngine, Image
2121
from .utils import execute_cmd
2222

23+
# The DOCKER_HOST environment variable is used by Docker to set a remote host.
24+
# The same DOCKER_HOST environment variable is also used
25+
# by the official Software Development Kit (SDK) and other libraries
26+
# as a way to select the socket to communicate with.
27+
# Because repo2docker uses the official Docker SDK,
28+
# Podman users MUST configure the DOCKER_HOST environment variable.
29+
DOCKER_HOST = os.getenv("DOCKER_HOST")
30+
# It is possible for a user to create a Podman socket
31+
# that does not include the string "podman"
32+
# but expect the string "podman" is good enough for repo2docker use case.
33+
if DOCKER_HOST is not None and DOCKER_HOST.find("podman") != -1:
34+
DOCKER_CLI = "podman"
35+
else:
36+
DOCKER_CLI = "docker"
37+
2338

2439
class DockerContainer(Container):
2540
def __init__(self, container):
@@ -66,6 +81,42 @@ class DockerEngine(ContainerEngine):
6681

6782
string_output = True
6883

84+
_container_cli = None
85+
86+
@property
87+
def container_cli(self):
88+
if self._container_cli is not None:
89+
return self._container_cli
90+
91+
cli = DOCKER_CLI
92+
93+
cli_version_call = [cli, "version"]
94+
try:
95+
docker_version = subprocess.run(
96+
cli_version_call, stdout=subprocess.DEVNULL, check=True
97+
)
98+
except (subprocess.CalledProcessError, OSError) as e:
99+
raise RuntimeError(
100+
f"The {cli} commandline client must be installed: {e}"
101+
) from None
102+
103+
# docker buildx is based in a plugin that might not be installed
104+
# https://github.com/docker/buildx
105+
#
106+
# podman buildx command is an alias of podman build.
107+
# Not all buildx build features are available in Podman.
108+
cli_buildx_version_call = [cli, "buildx", "version"]
109+
try:
110+
docker_buildx_version = subprocess.run(
111+
cli_buildx_version_call, stdout=subprocess.DEVNULL, check=True
112+
)
113+
except (subprocess.CalledProcessError, OSError) as e:
114+
raise RuntimeError(f"The buildx plugin must be installed: {e}") from None
115+
116+
self._container_cli = cli
117+
118+
return self._container_cli
119+
69120
extra_init_args = Dict(
70121
{},
71122
help="""
@@ -105,16 +156,7 @@ def build(
105156
platform=None,
106157
**kwargs,
107158
):
108-
if not shutil.which("docker"):
109-
raise RuntimeError("The docker commandline client must be installed")
110-
111-
# docker buildx is based in a plugin that might not be installed
112-
# https://github.com/docker/buildx
113-
docker_buildx_version = subprocess.run(["docker", "buildx", "version"])
114-
if docker_buildx_version.returncode:
115-
raise RuntimeError("The docker buildx plugin must be installed")
116-
117-
args = ["docker", "buildx", "build", "--progress", "plain"]
159+
args = [self.container_cli, "buildx", "build", "--progress", "plain"]
118160
if load:
119161
if push:
120162
raise ValueError(
@@ -171,14 +213,22 @@ def inspect_image(self, image):
171213
Return image configuration if it exists, otherwise None
172214
"""
173215
proc = subprocess.run(
174-
["docker", "image", "inspect", image], capture_output=True
216+
[self.container_cli, "image", "inspect", image], capture_output=True
175217
)
176218

177219
if proc.returncode != 0:
178220
return None
179221

180222
config = json.loads(proc.stdout.decode())[0]
181-
return Image(tags=config["RepoTags"], config=config["Config"])
223+
tags = config["RepoTags"]
224+
oci_image_configuration = config["Config"]
225+
226+
# WorkingDir is optional but docker always include it.
227+
# https://github.com/containers/podman/discussions/27313
228+
if "WorkingDir" not in oci_image_configuration:
229+
oci_image_configuration["WorkingDir"] = ""
230+
231+
return Image(tags=tags, config=oci_image_configuration)
182232

183233
@contextmanager
184234
def docker_login(self, username, password, registry):
@@ -200,7 +250,7 @@ def docker_login(self, username, password, registry):
200250
try:
201251
subprocess.run(
202252
[
203-
"docker",
253+
self.container_cli,
204254
"login",
205255
"--username",
206256
username,

tests/conftest.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,13 @@
2727
import requests
2828
import yaml
2929

30+
import docker
3031
from repo2docker.__main__ import make_r2d
3132

3233
TESTS_DIR = os.path.abspath(os.path.dirname(__file__))
3334

35+
DOCKER_CLIENT = client = docker.from_env()
36+
3437

3538
def pytest_collect_file(parent, file_path):
3639
if file_path.name == "verify":
@@ -90,6 +93,11 @@ def build_noop():
9093
# stop the container
9194
container.stop()
9295
app.wait_for_container(container)
96+
try:
97+
container.remove()
98+
except:
99+
pass
100+
DOCKER_CLIENT.images.remove(image=app.output_image_spec)
93101

94102
return test
95103

0 commit comments

Comments
 (0)