feat(reliability): integrate the ryuk container for better container cleanup (testcontainers#314)

> [!NOTE]
> Editor's note from @totallyzen:
> After a thorough discussion between @santi @alexanderankin, @kiview
and @totallyzen on the
[slack](https://testcontainers.slack.com/archives/C04SRG5AXNU/p1710156743640249)
we've decided this isn't a breaking change in api, but a modification in
behaviour. Therefore not worth a 5.0 but we'll release it under 4.x
> If this did end up breaking your workflow, come talk to us about your
use-case!

**What are you trying to do?**

Use Ryuk as the default resource cleanup strategy.

**Why should it be done this way?**

The current implementation of tc-python does not handle container
lifecycle management the same way as other TC implementations. In this
PR I introduce Ryuk to be the default resource cleanup strategy, in
order to be better aligned with other TC implementations.

Ryuk is enabled by default, but can be disabled with the
`TESTCONTAINERS_RYUK_DISABLED` env variable (which follows the behavior
of other TC implementations). Ryuk behavior can further be altered with
the `TESTCONTAINERS_RYUK_PRIVILEGED`, `RYUK_CONTAINER_IMAGE` and
`TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE` (also following the same
conventions from other implementations). Documentation of these env
variables is added to the README in the root of the repo.

The implementation uses a singleton container that starts an instance of
Ryuk and stores it on class / module level, so that it is reused within
a process. This follows the same pattern as in other languages, and Ryuk
behaviour itself is implemented the exact same way as other
implementations.

**BREAKING CHANGE**

`__del__` support is now removed, in order to better align with other TC
implementations. From the comments in the `__del__` implementation, it
seems like this was added as a final attempt to cleanup resources on
exit/garbage collection. This leaves three ways to cleanup resources,
which has better defined behaviors and follows modern patterns for
resource management in Python:

Method 1: Context Manager
Init/cleanup through a context manager (__enter__/__exit__):
```
with DockerContainer("postgres") as container:
    # some logic here

# end of context: container is killed by `__exit__` method
```

Method 2: Manual start() and stop()
```
container = DockerContainer("postgres").start()
# some logic here
container.stop()
```

Method 3: Ryuk
```
container = DockerContainer("postgres").start()
# some logic here

# You forget to .stop() the container, but Ryuk kills it for you 10 seconds after your process exits.
```

_Why remove `__del__`?_ According to the previous maintainer of the
repo, it has been causing “[a bunch of
issues](https://github.com/testcontainers/testcontainers-python/pull/314#discussion_r1185321083)”,
which I have personally experienced while using TC in a Django app, due
to the automatic GC behavior when no longer referencing the container
with a variable. E.g. if you instantiate the container in a method, only
returning the connection string, the Python garbage collector will
automatically call `__del__` on the instance at the end of the function,
thus killing your container. This leads to clunky workarounds like
having to store a reference to the container in a module-level variable,
or always having to return a reference to the container from the
function creating the container. In addition, the gc behaviour is not
consistent across Python implementations, making the reliance on
`__del__` flaky at best.

Also, having the __del__ method cleanup your container prevents us from
implementing `with_reuse()` (which is implemented in other TC
implementations) in the future, as a process exit would always instantly
kill the container, preventing us to use it in another process before
Ryuk reaps it.

**Next steps**

Once this PR is accepted, my plan is to implement the `with_reuse()`
functionality seen in other implementations, to enable faster / instant
usage of existing containers. This is very useful in simple testing
scenarios or local development workflows using hot reload behaviour. The
`with_reuse()` API requires the removal of `__del__` cleanup, as
otherwise the container would not be available for reuse due to the GC
reaping the container as soon as the process exits.

**Other changes**
- Adds “x-tc-sid=SESSION_ID” header to the underlying Docker API client
with the value of the current session ID (created on module init), in
order to enable Testcontainers Cloud to operate in “Turbo mode”
testcontainers#314 (comment)
- Adds labels `org.testcontainers.lang=python` and
`org.testcontainers.session-id=SESSION_ID`to the containers created by
TC
- As mentioned above, the env variables TESTCONTAINERS_RYUK_DISABLED,
TESTCONTAINERS_RYUK_PRIVILEGED, RYUK_CONTAINER_IMAGE and
TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE are now used for customizing
tc-python behavior.

---------

Co-authored-by: Andre Hofmeister <[email protected]>
Co-authored-by: Balint Bartha <[email protected]>

Author: 3 people

INDEX.rst

@@ -92,6 +92,21 @@ When trying to launch a testcontainer from within a Docker container, e.g., in c
 1. The container has to provide a docker client installation. Either use an image that has docker pre-installed (e.g. the `official docker images <https://hub.docker.com/_/docker>`_) or install the client from within the `Dockerfile` specification.
 2. The container has to have access to the docker daemon which can be achieved by mounting `/var/run/docker.sock` or setting the `DOCKER_HOST` environment variable as part of your `docker run` command.
 
+Configuration
+-------------
+
++-------------------------------------------+-------------------------------+------------------------------------------+
+| Env Variable                              | Example                       | Description                              |
++===========================================+===============================+==========================================+
+| ``TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE`` | ``/var/run/docker.sock``      | Path to Docker's socket used by ryuk     |
++-------------------------------------------+-------------------------------+------------------------------------------+
+| ``TESTCONTAINERS_RYUK_PRIVILEGED``        | ``false``                     | Run ryuk as a privileged container       |
++-------------------------------------------+-------------------------------+------------------------------------------+
+| ``TESTCONTAINERS_RYUK_DISABLED``          | ``false``                     | Disable ryuk                             |
++-------------------------------------------+-------------------------------+------------------------------------------+
+| ``RYUK_CONTAINER_IMAGE``                  | ``testcontainers/ryuk:0.5.1`` | Custom image for ryuk                    |
++-------------------------------------------+-------------------------------+------------------------------------------+
+
 Development and Contributing
 ----------------------------
 

README.md

@@ -22,3 +22,12 @@ For more information, see [the docs][readthedocs].
 ```
 
 The snippet above will spin up a postgres database in a container. The `get_connection_url()` convenience method returns a `sqlalchemy` compatible url we use to connect to the database and retrieve the database version.
+
+## Configuration
+
+| Env Variable                              | Example                       | Description                              |
+| ----------------------------------------- | ----------------------------- | ---------------------------------------- |
+| `TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE`   | `/var/run/docker.sock`        | Path to Docker's socket used by ryuk     |
+| `TESTCONTAINERS_RYUK_PRIVILEGED`          | `false`                       | Run ryuk as a privileged container       |
+| `TESTCONTAINERS_RYUK_DISABLED`            | `false`                       | Disable ryuk                             |
+| `RYUK_CONTAINER_IMAGE`                    | `testcontainers/ryuk:0.5.1`   | Custom image for ryuk                    |

core/testcontainers/core/config.py

@@ -3,3 +3,8 @@
 MAX_TRIES = int(environ.get("TC_MAX_TRIES", 120))
 SLEEP_TIME = int(environ.get("TC_POOLING_INTERVAL", 1))
 TIMEOUT = MAX_TRIES * SLEEP_TIME
+
+RYUK_IMAGE: str = environ.get("RYUK_CONTAINER_IMAGE", "testcontainers/ryuk:0.5.1")
+RYUK_PRIVILEGED: bool = environ.get("TESTCONTAINERS_RYUK_PRIVILEGED", "false") == "true"
+RYUK_DISABLED: bool = environ.get("TESTCONTAINERS_RYUK_DISABLED", "false") == "true"
+RYUK_DOCKER_SOCKET: str = environ.get("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE", "/var/run/docker.sock")

core/testcontainers/core/container.py

@@ -1,13 +1,16 @@
-import contextlib
 from platform import system
-from typing import Optional
-
-from docker.models.containers import Container
+from socket import socket
+from typing import TYPE_CHECKING, Optional
 
+from testcontainers.core.config import RYUK_DISABLED, RYUK_DOCKER_SOCKET, RYUK_IMAGE, RYUK_PRIVILEGED
 from testcontainers.core.docker_client import DockerClient
 from testcontainers.core.exceptions import ContainerStartException
+from testcontainers.core.labels import LABEL_SESSION_ID, SESSION_ID
 from testcontainers.core.utils import inside_container, is_arm, setup_logger
-from testcontainers.core.waiting_utils import wait_container_is_ready
+from testcontainers.core.waiting_utils import wait_container_is_ready, wait_for_logs
+
+if TYPE_CHECKING:
+    from docker.models.containers import Container
 
 logger = setup_logger(__name__)
 
@@ -25,7 +28,12 @@ class DockerContainer:
         ...    delay = wait_for_logs(container, "Hello from Docker!")
     """
 
-    def __init__(self, image: str, docker_client_kw: Optional[dict] = None, **kwargs) -> None:
+    def __init__(
+        self,
+        image: str,
+        docker_client_kw: Optional[dict] = None,
+        **kwargs,
+    ) -> None:
         self.env = {}
         self.ports = {}
         self.volumes = {}
@@ -58,7 +66,10 @@ def maybe_emulate_amd64(self) -> "DockerContainer":
             return self.with_kwargs(platform="linux/amd64")
         return self
 
-    def start(self) -> "DockerContainer":
+    def start(self):
+        if not RYUK_DISABLED and self.image != RYUK_IMAGE:
+            logger.debug("Creating Ryuk container")
+            Reaper.get_instance()
         logger.info("Pulling image %s", self.image)
         docker_client = self.get_docker_client()
         self._container = docker_client.run(
@@ -69,7 +80,7 @@ def start(self) -> "DockerContainer":
             ports=self.ports,
             name=self._name,
             volumes=self.volumes,
-            **self._kwargs
+            **self._kwargs,
         )
         logger.info("Container started: %s", self._container.short_id)
         return self
@@ -78,21 +89,12 @@ def stop(self, force=True, delete_volume=True) -> None:
         self._container.remove(force=force, v=delete_volume)
         self.get_docker_client().client.close()
 
-    def __enter__(self) -> "DockerContainer":
+    def __enter__(self):
         return self.start()
 
     def __exit__(self, exc_type, exc_val, exc_tb) -> None:
         self.stop()
 
-    def __del__(self) -> None:
-        """
-        __del__ runs when Python attempts to garbage collect the object.
-        In case of leaky test design, we still attempt to clean up the container.
-        """
-        with contextlib.suppress(Exception):
-            if self._container is not None:
-                self.stop()
-
     def get_container_host_ip(self) -> str:
         # infer from docker host
         host = self.get_docker_client().host()
@@ -140,7 +142,7 @@ def with_volume_mapping(self, host: str, container: str, mode: str = "ro") -> "D
         self.volumes[host] = mapping
         return self
 
-    def get_wrapped_container(self) -> Container:
+    def get_wrapped_container(self) -> "Container":
         return self._container
 
     def get_docker_client(self) -> DockerClient:
@@ -155,3 +157,54 @@ def exec(self, command) -> tuple[int, str]:
         if not self._container:
             raise ContainerStartException("Container should be started before executing a command")
         return self._container.exec_run(command)
+
+
+class Reaper:
+    _instance: "Optional[Reaper]" = None
+    _container: Optional[DockerContainer] = None
+    _socket: Optional[socket] = None
+
+    @classmethod
+    def get_instance(cls) -> "Reaper":
+        if not Reaper._instance:
+            Reaper._instance = Reaper._create_instance()
+
+        return Reaper._instance
+
+    @classmethod
+    def delete_instance(cls) -> None:
+        if Reaper._socket is not None:
+            Reaper._socket.close()
+            Reaper._socket = None
+
+        if Reaper._container is not None:
+            Reaper._container.stop()
+            Reaper._container = None
+
+        if Reaper._instance is not None:
+            Reaper._instance = None
+
+    @classmethod
+    def _create_instance(cls) -> "Reaper":
+        logger.debug(f"Creating new Reaper for session: {SESSION_ID}")
+
+        Reaper._container = (
+            DockerContainer(RYUK_IMAGE)
+            .with_name(f"testcontainers-ryuk-{SESSION_ID}")
+            .with_exposed_ports(8080)
+            .with_volume_mapping(RYUK_DOCKER_SOCKET, "/var/run/docker.sock", "rw")
+            .with_kwargs(privileged=RYUK_PRIVILEGED)
+            .start()
+        )
+        wait_for_logs(Reaper._container, r".* Started!")
+
+        container_host = Reaper._container.get_container_host_ip()
+        container_port = int(Reaper._container.get_exposed_port(8080))
+
+        Reaper._socket = socket()
+        Reaper._socket.connect((container_host, container_port))
+        Reaper._socket.send(f"label={LABEL_SESSION_ID}={SESSION_ID}\r\n".encode())
+
+        Reaper._instance = Reaper()
+
+        return Reaper._instance

core/testcontainers/core/docker_client.py

@@ -10,7 +10,6 @@
 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 #    License for the specific language governing permissions and limitations
 #    under the License.
-import atexit
 import functools as ft
 import os
 import urllib
@@ -19,10 +18,10 @@
 from typing import Optional, Union
 
 import docker
-from docker.errors import NotFound
 from docker.models.containers import Container, ContainerCollection
 
-from .utils import default_gateway_ip, inside_container, setup_logger
+from testcontainers.core.labels import SESSION_ID, create_labels
+from testcontainers.core.utils import default_gateway_ip, inside_container, setup_logger
 
 LOGGER = setup_logger(__name__)
 TC_FILE = ".testcontainers.properties"
@@ -42,6 +41,7 @@ def __init__(self, **kwargs) -> None:
             self.client = docker.DockerClient(base_url=docker_host)
         else:
             self.client = docker.from_env(**kwargs)
+        self.client.api.headers["x-tc-sid"] = SESSION_ID
 
     @ft.wraps(ContainerCollection.run)
     def run(
@@ -50,6 +50,7 @@ def run(
         command: Optional[Union[str, list[str]]] = None,
         environment: Optional[dict] = None,
         ports: Optional[dict] = None,
+        labels: Optional[dict[str, str]] = None,
         detach: bool = False,
         stdout: bool = True,
         stderr: bool = False,
@@ -65,10 +66,9 @@ def run(
             detach=detach,
             environment=environment,
             ports=ports,
+            labels=create_labels(image, labels),
             **kwargs,
         )
-        if detach:
-            atexit.register(_stop_container, container)
         return container
 
     def port(self, container_id: str, port: int) -> int:
@@ -145,12 +145,3 @@ def read_tc_properties() -> dict[str, str]:
             tuples = [line.split("=") for line in contents.readlines() if "=" in line]
             settings = {**settings, **{item[0].strip(): item[1].strip() for item in tuples}}
     return settings
-
-
-def _stop_container(container: Container) -> None:
-    try:
-        container.stop()
-    except NotFound:
-        pass
-    except Exception as ex:
-        LOGGER.warning("failed to shut down container %s with image %s: %s", container.id, container.image, ex)

core/testcontainers/core/labels.py

@@ -0,0 +1,20 @@
+from typing import Optional
+from uuid import uuid4
+
+from testcontainers.core.config import RYUK_IMAGE
+
+SESSION_ID: str = str(uuid4())
+LABEL_SESSION_ID = "org.testcontainers.session-id"
+LABEL_LANG = "org.testcontainers.lang"
+
+
+def create_labels(image: str, labels: Optional[dict[str, str]]) -> dict[str, str]:
+    if labels is None:
+        labels = {}
+    labels[LABEL_LANG] = "python"
+
+    if image == RYUK_IMAGE:
+        return labels
+
+    labels[LABEL_SESSION_ID] = SESSION_ID
+    return labels

core/tests/test_ryuk.py

@@ -0,0 +1,24 @@
+from testcontainers.core import container
+from testcontainers.core.container import Reaper
+from testcontainers.core.container import DockerContainer
+from testcontainers.core.waiting_utils import wait_for_logs
+
+
+def test_wait_for_reaper():
+    container = DockerContainer("hello-world").start()
+    wait_for_logs(container, "Hello from Docker!")
+
+    assert Reaper._socket is not None
+    Reaper._socket.close()
+
+    assert Reaper._container is not None
+    wait_for_logs(Reaper._container, r".* Removed \d .*", timeout=30)
+
+    Reaper.delete_instance()
+
+
+def test_container_without_ryuk(monkeypatch):
+    monkeypatch.setattr(container, "RYUK_DISABLED", True)
+    with DockerContainer("hello-world") as cont:
+        wait_for_logs(cont, "Hello from Docker!")
+        assert Reaper._instance is None

pyproject.toml

@@ -199,6 +199,9 @@ ignore = [
     "INP001"
 ]
 
+[tool.ruff.lint.pyupgrade]
+keep-runtime-typing = true
+
 [tool.ruff.lint.flake8-type-checking]
 strict = true