Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4ae4be0
Add support to DockerEngine
rgaiacs Oct 1, 2025
3108885
Use "container" instead of Docker when possible in index.md
rgaiacs Oct 1, 2025
45c3fb3
Use "container" instead of Docker when possible in start.md
rgaiacs Oct 1, 2025
5f17fed
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 1, 2025
606cc87
Auto configure container_cli property based on DOCKER_HOST
rgaiacs Oct 6, 2025
d4fc92d
Add podman as container cli for GitHub Actions
rgaiacs Oct 6, 2025
d1985fb
Select correct hostname for Podman
rgaiacs Oct 6, 2025
905fe0b
Add note about Podman service to documentation
rgaiacs Oct 17, 2025
080f61b
Enable Podman socket during test
rgaiacs Oct 17, 2025
916d3c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 17, 2025
b0ad00b
Avoid infinite loop in test
rgaiacs Oct 17, 2025
f05f420
Hide output of "docker version" test
rgaiacs Oct 17, 2025
8d74803
Add WorkingDir to bridge the gap with Podman
rgaiacs Oct 17, 2025
7ac468a
Fix tests/unit/test_app.py::test_extra_buildx_build_args
rgaiacs Oct 30, 2025
3027e2c
Make DOCKER_HOST more robust
rgaiacs Nov 3, 2025
e1807c5
Prefer to use attribute over internal variable
rgaiacs Nov 3, 2025
07808e6
Minor improve to docker.py wrap
rgaiacs Nov 4, 2025
f29e7de
Expose DOCKER_CLI from docker.py
rgaiacs Nov 5, 2025
f73d267
Mark some tests with @pytest.mark.skipif
rgaiacs Nov 5, 2025
ba52252
Skip registry test for Podman
rgaiacs Nov 5, 2025
7063a1f
Fix English mistakes in documentation
rgaiacs Nov 13, 2025
308ac22
Reduce the number of tests for Podman
rgaiacs Nov 13, 2025
37d2612
Add comment explaining DOCKER_HOST
rgaiacs Dec 4, 2025
fee0b11
Guard subprocess.run() with try block
rgaiacs Dec 4, 2025
9259ca8
Re-write matrix strategy to use include
rgaiacs Dec 4, 2025
7270129
Remove if-block already cover in try-block
rgaiacs Dec 5, 2025
7f6b65e
Fix include for GItHub Actions
rgaiacs Jan 8, 2026
6baaf08
Delete container and image after test
rgaiacs Jan 13, 2026
f010e36
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 13, 2026
9032292
Disable test of conda for podman
rgaiacs Jan 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ jobs:
matrix:
ubuntu_version: ["24.04"]
python_version: ["3.13"]
container_cli:
- docker
repo_type:
- base
- conda
Expand All @@ -68,11 +70,34 @@ jobs:
- norun
# Playwright test
- ui
# If the item to be include would overwrite a value,
# it must have all the fields!
include:
# The earliest actions/setup-python versions depend on the runner.
- ubuntu_version: "24.04"
python_version: "3.10"
container_cli: docker
repo_type: venv
# Run all tests for both Docker and Podman is expensive.
# We only run the "smoke" test for Podman.
- ubuntu_version: "24.04"
python_version: "3.13"
container_cli: podman
repo_type: base
# We are observing "no space left on device" on GitHub Actions,
# https://github.com/jupyterhub/repo2docker/pull/1471#issuecomment-3744990787
# - ubuntu_version: "24.04"
# python_version: "3.13"
# container_cli: podman
# repo_type: conda
- ubuntu_version: "24.04"
python_version: "3.13"
container_cli: podman
repo_type: dockerfile
- ubuntu_version: "24.04"
python_version: "3.13"
container_cli: podman
repo_type: unit

steps:
- uses: actions/checkout@v6
Expand Down Expand Up @@ -104,6 +129,11 @@ jobs:

- name: Run pytest
run: |
if [ "${{ matrix.container_cli }}" = "podman" ]
then
systemctl --user start podman.socket
export DOCKER_HOST=unix://$(podman info --format '{{.Host.RemoteSocket.Path}}')
fi
pytest --verbose --color=yes --durations=10 --cov=repo2docker tests/${{ matrix.repo_type }}

- uses: codecov/codecov-action@v5
16 changes: 10 additions & 6 deletions docs/source/index.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,19 @@
# Welcome to `repo2docker`'s documentation

```{important}
Despite the name, `repo2docker` can be used by container technology other than [Docker](https://docs.docker.com/engine/), for example [Podman](https://podman.io/).
```

`repo2docker` lets you **reproducibly build and run user environment container images for interactive computing and data workflows from source code repositories**. Optionally, the container image can be pushed to a Docker registry.

Also, `repo2docker` is the tool used to build container images for [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) and the tool used by [BinderHub](https://binderhub.readthedocs.io) to build images on demand.

::::{grid}
:::{grid-item-card} 🔧 Build reproducible data science environments from repositories
Build a reproducible data science environment as a Docker image and execute code interactively. Use many [configuration files](#config-files) to control language, tools, and setup instructions.
Build a reproducible data science environment as a container image and execute code interactively. Use many [configuration files](#config-files) to control language, tools, and setup instructions.
:::
:::{grid-item-card} 🚀 Deploy environments in JupyterHub or Binder
Push environment images to a Docker registry for re-use in data science environment services like [JupyterHub](https://jupyterhub.readthedocs.io) or [a Binder instance](https://mybinder.org), or for other communities to build upon your base environment.
Push environment images to a container registry for re-use in data science environment services like [JupyterHub](https://jupyterhub.readthedocs.io) or [a Binder instance](https://mybinder.org), or for other communities to build upon your base environment.
:::
:::{grid-item-card} ☁️ Host repositories in many providers
Host repositories in: a Git server like [GitHub](https://github.com/) or [GitLab](https://gitlab.com/), an open science repository like [Zenodo](https://zenodo.org) or [Figshare](https://figshare.com), a hosted data platform like a [Dataverse installation](https://dataverse.org/), an archive like the
Expand All @@ -19,7 +23,7 @@ Host repositories in: a Git server like [GitHub](https://github.com/) or [GitLab

## What is a user environment container image and why would I build one with `repo2docker`?

A **user environment container image** contains the entire software environment that a user may access from an interactive data science session. For example, it might contain many **programming languages**, **software for data analysis**, or even **content files and datasets** available to anybody that accesses that environment. Container images are built with [Docker](https://www.docker.com/), a standard open source tool for defining, building, and deploying images.
A **user environment container image** contains the entire software environment that a user may access from an interactive data science session. For example, it might contain many **programming languages**, **software for data analysis**, or even **content files and datasets** available to anybody that accesses that environment. Container images are built in accordance with the spectifications published by the [Open Container Initiative](https://opencontainers.org/).

Many data science platforms and services like [JupyterHub](https://jupyterhub.readthedocs.io) and [Binder](https://mybinder.org) launch interactive data science sessions **with a user environment container image attached**, meaning that the user gains access to whatever is in the container image. In short, this allows somebody to define and build the user image one time, in a way that users can reproducibly re-use many times.

Expand All @@ -43,9 +47,9 @@ repo2docker <source-repository>
It performs these steps:

1. Inspects the repository for [configuration files](#config-files). These will be used to build the environment needed to run the repository.
2. Builds a Docker image with an environment specified in these [configuration files](#config-files).
2. Builds a container image with an environment specified in these [configuration files](#config-files).
3. Runs the image to let you explore the repository interactively via Jupyter notebooks, RStudio, or many other interfaces (this is optional).
4. Pushes the images to a Docker registry so that it may be accessed remotely (this is optional).
4. Pushes the images to a container registry so that it may be accessed remotely (this is optional).

[swhid]: https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html

Expand All @@ -55,7 +59,7 @@ Please report [bugs](https://github.com/jupyterhub/repo2docker/issues),

## Get started with `repo2docker`

This tutorial walks you through setting up `repo2docker`, building your first environment image, and running it locally with Docker.
This tutorial walks you through setting up `repo2docker`, building your first environment image, and running it locally with a container engine.

```{toctree}
:maxdepth: 2
Expand Down
71 changes: 48 additions & 23 deletions docs/source/start.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,47 +2,72 @@

This tutorial guides you through installing `repo2docker` and building your first environment image.

(install)=
## Prerequisite

## Install `repo2docker`
### Python

`repo2docker` requires Python 3.6 or above on Linux and macOS.
`repo2docker` requires Python 3.6 or above.

:::{admonition} Windows support is experimental
### Container Engine

This [article about using Windows and the WSL](https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly) (Windows Subsystem for Linux or
Bash on Windows) provides additional information about Windows and Docker.
:::
`repo2docker` requires a container engine compatible with the specification published by the [Open Container Initiative](https://opencontainers.org/).

### Prerequisite: Install Docker
#### Docker

Install [Docker](https://www.docker.com), as it is required to build Docker images.
The [Community Edition](https://docs.docker.com/install/) is available for free.
```{important}
Only the [Docker Engine](https://docs.docker.com/engine/) is open source. [Docker Desktop](https://docs.docker.com/get-started/get-docker/) requires a license.
```

Recent versions of Docker are recommended.
Follow [Docker's official installation steps](https://docs.docker.com/get-started/get-docker/).

### Install `repo2docker` with `pip`
#### Podman

```{warning}
The name of the package on [PyPI](https://pypi.org/) is [`jupyter-repo2docker`](https://pypi.org/project/jupyter-repo2docker/) instead of `repo2docker`.
```
Follow [Podman's official installation steps](https://podman.io/docs/installation).

We recommend installing `repo2docker` with the `pip` tool:
After completing the installation of Podman,

```
1. create a [listening service for Podman](https://docs.podman.io/en/latest/markdown/podman-system-service.1.html) by running

```bash
systemctl --user start podman.socket
```

1. configure the `DOCKER_HOST` environment variable following [Podman's official procedure](https://podman-desktop.io/docs/migrating-from-docker/using-the-docker_host-environment-variable#procedure). You might want to configure the `DOCKER_HOST` environment variable to persist in your `~/.bashrc`.

(install)=

## Install `repo2docker`

### Install `repo2docker` with `pip`

It is recommend to install `repo2docker` with the `pip` tool:

```bash
python3 -m pip install jupyter-repo2docker
```

(usage)=

## Build a repository with `repo2docker`

Now that you've installed Docker and `repo2docker`, we can build a repository.
To do so, follow these steps.
Now that you've installed a container engine and `repo2docker`, you can build a repository.
To do so, continue following this guide.

### Start the container engine

### Start Docker
Ensure that the container engine is running.

Follow the [instructions for starting Docker](https://docs.docker.com/engine/daemon/start/) to start a Docker process.
#### Docker

Follow the [offcial instructions for starting Docker](https://docs.docker.com/engine/daemon/start/).

#### Podman

Run

```bash
podman info
```

### Build an image from a URL

Expand All @@ -55,12 +80,12 @@ jupyter-repo2docker https://github.com/binder-examples/requirements
You'll see `repo2docker` take the following actions:

1. Inspect the repository for [configuration files](#config-files). It will detect the `requirements.txt` file in the repository.
2. Build a Docker image using the configuration files. In this case, the `requirements.txt` file will correspond to a Python environment.
2. Build a container image using the configuration files. In this case, the `requirements.txt` file will correspond to a Python environment.
3. Run the image to let you explore the repository interactively.

Click the link provided and you'll be taken to an interactive Jupyter Notebook interface where you can run commands interactively inside the environment.

## Learn more

This is a simple example building an environment image for your repository.
To learn more about the kinds of source repositories, environments, and use-cases that repo2docker supports, see [the `repo2docker` user guide](./use/index.md).
To learn more about the kinds of source repositories, environments, and use-cases that `repo2docker` supports, see [the `repo2docker` user guide](./use/index.md).
8 changes: 6 additions & 2 deletions repo2docker/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,11 @@ def start_container(self):

docker_host = os.environ.get("DOCKER_HOST")
if docker_host:
host_name = urlparse(docker_host).hostname
docker_host_parsed = urlparse(docker_host)
if docker_host_parsed.scheme == "unix":
host_name = "127.0.0.1"
else:
host_name = docker_host_parsed.hostname
else:
host_name = "127.0.0.1"
self.hostname = host_name
Expand Down Expand Up @@ -621,7 +625,7 @@ def start_container(self):
"notebook",
"--ip=0.0.0.0",
f"--port={container_port}",
f"--ServerApp.custom_display_url=http://{host_name}:{host_port}",
f"--ServerApp.custom_display_url=http://{self.hostname}:{self.port}",
"--ServerApp.default_url=/lab",
]
else:
Expand Down
78 changes: 64 additions & 14 deletions repo2docker/docker.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,28 @@
from pathlib import Path

from iso8601 import parse_date
from traitlets import Dict, List, Unicode
from traitlets import Dict, List, Unicode, default

import docker

from .engine import Container, ContainerEngine, Image
from .utils import execute_cmd

# The DOCKER_HOST environment variable is used by Docker to set a remote host.
# The same DOCKER_HOST environment variable is also used
# by the official Software Development Kit (SDK) and other libraries
# as a way to select the socket to communicate with.
# Because repo2docker uses the official Docker SDK,
# Podman users MUST configure the DOCKER_HOST environment variable.
DOCKER_HOST = os.getenv("DOCKER_HOST")
# It is possible for a user to create a Podman socket
# that does not include the string "podman"
# but expect the string "podman" is good enough for repo2docker use case.
if DOCKER_HOST is not None and DOCKER_HOST.find("podman") != -1:
DOCKER_CLI = "podman"
else:
DOCKER_CLI = "docker"


class DockerContainer(Container):
def __init__(self, container):
Expand Down Expand Up @@ -66,6 +81,42 @@ class DockerEngine(ContainerEngine):

string_output = True

_container_cli = None

@property
def container_cli(self):
if self._container_cli is not None:
return self._container_cli

cli = DOCKER_CLI

cli_version_call = [cli, "version"]
try:
docker_version = subprocess.run(
cli_version_call, stdout=subprocess.DEVNULL, check=True
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check=True will raise a CalledProcessError if the return code is not 0, so we can get rid of the if docker_version.returncode check, and similarly for the buildx check

)
except (subprocess.CalledProcessError, OSError) as e:
raise RuntimeError(
f"The {cli} commandline client must be installed: {e}"
) from None

# docker buildx is based in a plugin that might not be installed
# https://github.com/docker/buildx
#
# podman buildx command is an alias of podman build.
# Not all buildx build features are available in Podman.
cli_buildx_version_call = [cli, "buildx", "version"]
try:
docker_buildx_version = subprocess.run(
cli_buildx_version_call, stdout=subprocess.DEVNULL, check=True
)
except (subprocess.CalledProcessError, OSError) as e:
raise RuntimeError(f"The buildx plugin must be installed: {e}") from None

self._container_cli = cli

return self._container_cli

extra_init_args = Dict(
{},
help="""
Expand Down Expand Up @@ -105,16 +156,7 @@ def build(
platform=None,
**kwargs,
):
if not shutil.which("docker"):
raise RuntimeError("The docker commandline client must be installed")

# docker buildx is based in a plugin that might not be installed
# https://github.com/docker/buildx
docker_buildx_version = subprocess.run(["docker", "buildx", "version"])
if docker_buildx_version.returncode:
raise RuntimeError("The docker buildx plugin must be installed")

args = ["docker", "buildx", "build", "--progress", "plain"]
args = [self.container_cli, "buildx", "build", "--progress", "plain"]
if load:
if push:
raise ValueError(
Expand Down Expand Up @@ -171,14 +213,22 @@ def inspect_image(self, image):
Return image configuration if it exists, otherwise None
"""
proc = subprocess.run(
["docker", "image", "inspect", image], capture_output=True
[self.container_cli, "image", "inspect", image], capture_output=True
)

if proc.returncode != 0:
return None

config = json.loads(proc.stdout.decode())[0]
return Image(tags=config["RepoTags"], config=config["Config"])
tags = config["RepoTags"]
oci_image_configuration = config["Config"]

# WorkingDir is optional but docker always include it.
# https://github.com/containers/podman/discussions/27313
if "WorkingDir" not in oci_image_configuration:
oci_image_configuration["WorkingDir"] = ""

return Image(tags=tags, config=oci_image_configuration)

@contextmanager
def docker_login(self, username, password, registry):
Expand All @@ -200,7 +250,7 @@ def docker_login(self, username, password, registry):
try:
subprocess.run(
[
"docker",
self.container_cli,
"login",
"--username",
username,
Expand Down
8 changes: 8 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,13 @@
import requests
import yaml

import docker
from repo2docker.__main__ import make_r2d

TESTS_DIR = os.path.abspath(os.path.dirname(__file__))

DOCKER_CLIENT = client = docker.from_env()


def pytest_collect_file(parent, file_path):
if file_path.name == "verify":
Expand Down Expand Up @@ -90,6 +93,11 @@ def build_noop():
# stop the container
container.stop()
app.wait_for_container(container)
try:
container.remove()
except:
pass
DOCKER_CLIENT.images.remove(image=app.output_image_spec)

return test

Expand Down
Loading