Use bookworm; target specific Python & PostgreSQL (#1414)

Update dependencies to comply with Synapse's deprecation policy: https://element-hq.github.io/synapse/latest/deprecation_policy.html

- Upgrade to Debian Bookworm (the current oldstable) to pull in the oldest supported version of SQLite.
- Instead of installing PostgreSQL from Debian repositories, use the PostgreSQL Apt Repository to install its lowest-supported version.
- Instead of installing Python from Debian repositories, use uv to install its lowest-supported version.

Author: AndrewFerr

.github/workflows/docker.yml

@@ -23,10 +23,14 @@ jobs:
     strategy:
       matrix:
         include:
-          - base_image: debian:bullseye
-            tag: bullseye
-          - base_image: debian:testing
-            tag: testing
+          - tag: bookworm
+            build_args: |
+              BASE_IMAGE=debian:bookworm
+              POSTGRESQL_VERSION=13
+          - tag: testing
+            build_args: |
+              BASE_IMAGE=debian:testing
+              POSTGRESQL_VERSION=18
     steps:
       - name: Set up QEMU
         id: QEMU
@@ -53,7 +57,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           labels: "gitsha1=${{ github.sha }}"
           file: docker/base.Dockerfile
-          build-args: "BASE_IMAGE=${{ matrix.base_image }}"
+          build-args: ${{ matrix.build_args }}
           tags: matrixdotorg/sytest:${{ matrix.tag }}
 
   build-dependent-images:
@@ -63,16 +67,18 @@ jobs:
     strategy:
       matrix:
         include:
-          - sytest_image_tag: bullseye
+          - sytest_image_tag: bookworm
             dockerfile: synapse
-            tags: "matrixdotorg/sytest-synapse:bullseye"
-            build_args: "SYTEST_IMAGE_TAG=bullseye"
+            tags: "matrixdotorg/sytest-synapse:bookworm"
+            build_args: |
+              SYTEST_IMAGE_TAG=bookworm
+              PYTHON_VERSION=3.10
           - sytest_image_tag: testing
             dockerfile: synapse
             tags: "matrixdotorg/sytest-synapse:testing"
             build_args: |
               SYTEST_IMAGE_TAG=testing
-              SYSTEM_PIP_INSTALL_SUFFIX=--break-system-packages
+              PYTHON_VERSION=3.13
 
     steps:
       - name: Set up QEMU

.github/workflows/pipeline.yml

@@ -21,23 +21,26 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - label: Debian Bullseye (Py 3.9, SQLite) Monolith
-            sytest-tag: bullseye
+          - label: Debian Bookworm (Py 3.10, SQLite 3.40) Monolith
+            sytest-tag: bookworm
 
-          - label: Debian Bullseye (Py 3.9, PG 13) Monolith
-            sytest-tag: bullseye
+          - label: Debian Bookworm (Py 3.10, PG 13) Monolith
+            sytest-tag: bookworm
             postgres: postgres
 
-          - label: Debian Bullseye (Py 3.9, PG 13) Workers
-            sytest-tag: bullseye
+          - label: Debian Bookworm (Py 3.10, PG 13) Workers
+            sytest-tag: bookworm
             postgres: postgres
             workers: workers
 
-          - label: Debian Testing (Py 3.13, PG 17), Monolith
+          - label: Debian Testing (Py 3.13, SQLite 3.46) Monolith
+            sytest-tag: testing
+
+          - label: Debian Testing (Py 3.13, PG 18), Monolith
             sytest-tag: testing
             postgres: postgres
 
-          - label: Debian Testing (Py 3.13, PG 17), Workers
+          - label: Debian Testing (Py 3.13, PG 18), Workers
             sytest-tag: testing
             postgres: postgres
             workers: workers

docker/README.md

@@ -8,9 +8,9 @@ but its dependencies are.
 Included currently is:
 
 - `matrixdotorg/sytest` Base container with SyTest dependencies installed
-  - Tagged by underlying Debian/Ubuntu image: `bullseye` or `testing`
+  - Tagged by underlying Debian/Ubuntu image: `bookworm` or `testing`
 - `matrixdotorg/sytest-synapse`: Runs SyTest against Synapse
-  - Tagged by underlying Debian/Ubuntu image: `bullseye` or `testing`
+  - Tagged by underlying Debian/Ubuntu image: `bookworm` or `testing`
 
 ## Target-specific details
 
@@ -23,7 +23,7 @@ it is useful to mount a volume there too.
 For example:
 
 ```
-docker run --rm -it -v /path/to/synapse\:/src:ro -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:bullseye
+docker run --rm -it -v /path/to/synapse\:/src:ro -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:bookworm
 ```
 
 The following environment variables can be set with `-e` to control the test run:
@@ -47,7 +47,7 @@ An example of running Synapse in worker mode:
 
 ```
 docker run --rm -it -e POSTGRES=1 -e WORKERS=1 -v /path/to/synapse\:/src:ro \
-    -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:bullseye
+    -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:bookworm
 ```
 
 ### Dendrite
@@ -72,7 +72,7 @@ for example:
 docker run --rm -it \
     -e SYTEST_BRANCH="my-sytest-branch"
     -v /path/to/synapse\:/src:ro -v /path/to/where/you/want/logs\:/logs
-    matrixdotorg/sytest-synapse:bullseye
+    matrixdotorg/sytest-synapse:bookworm
 ```
 
 If the branch referred to by `SYTEST_BRANCH` does not exist, `develop` is used
@@ -84,7 +84,7 @@ the container:
 
 ```
 docker run --rm -it -v /path/to/synapse\:/src:ro -v /path/to/where/you/want/logs\:/logs \
-    -v /path/to/code/sytest\:/sytest:ro matrixdotorg/sytest-synapse:bullseye
+    -v /path/to/code/sytest\:/sytest:ro matrixdotorg/sytest-synapse:bookworm
 ```
 
 ## Running a single test file, and other sytest commandline options
@@ -93,7 +93,7 @@ You can pass arguments to sytest by adding them at the end of the
 docker command. For example:
 
 ```
-docker run --rm -it ... matrixdotorg/sytest-synapse:bullseye tests/20profile-events.pl
+docker run --rm -it ... matrixdotorg/sytest-synapse:bookworm tests/20profile-events.pl
 ```
 
 ## Building the containers

docker/base.Dockerfile

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=debian:bullseye
+ARG BASE_IMAGE=debian:bookworm
 
 FROM ${BASE_IMAGE}
 
@@ -20,14 +20,23 @@ RUN apt-get -qq update && apt-get -qq install -y \
     libz-dev \
     locales \
     perl \
-    postgresql \
+    postgresql-common \
     rsync \
     sqlite3 \
     wget \
     libicu-dev \
     pkg-config \
     && rm -rf /var/lib/apt/lists/*
 
+# Set the default PostgreSQL version to the minimum supported by Synapse:
+# https://element-hq.github.io/synapse/latest/deprecation_policy.html
+ARG POSTGRESQL_VERSION=13
+# Install a specific PostgreSQL version via the official upstream repository:
+# https://wiki.debian.org/PostgreSql%20%20#PGDG_Repository
+RUN /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y \
+    && apt -qq install -y postgresql-${POSTGRESQL_VERSION} \
+    && rm -rf /var/lib/apt/lists/*
+
 # Set up the locales, as the default Debian image only has C, and PostgreSQL needs the correct locales to make a UTF-8 database
 RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
     dpkg-reconfigure --frontend=noninteractive locales && \

docker/dendrite.Dockerfile

@@ -1,4 +1,4 @@
-ARG SYTEST_IMAGE_TAG=bullseye
+ARG SYTEST_IMAGE_TAG=bookworm
 FROM matrixdotorg/sytest:${SYTEST_IMAGE_TAG}
 
 ARG GO_VERSION=1.19.1

docker/synapse.Dockerfile

@@ -1,18 +1,15 @@
-ARG SYTEST_IMAGE_TAG=bullseye
+ARG SYTEST_IMAGE_TAG=bookworm
 
 FROM matrixdotorg/sytest:${SYTEST_IMAGE_TAG}
 
-ARG PYTHON_VERSION=python3
-ARG SYSTEM_PIP_INSTALL_SUFFIX=""
-
 ENV DEBIAN_FRONTEND noninteractive
 
 # Ensure we die correctly when using pipes in RUN.
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 
 RUN apt-get -qq update && apt-get -qq install -y \
-        apt-utils ${PYTHON_VERSION} ${PYTHON_VERSION}-dev ${PYTHON_VERSION}-venv \
-        python3-pip eatmydata redis-server curl
+        apt-utils eatmydata redis-server curl \
+        && rm -rf /var/lib/apt/lists/*
 
 ENV RUSTUP_HOME=/rust
 ENV CARGO_HOME=/cargo
@@ -21,10 +18,15 @@ RUN mkdir /rust /cargo
 
 RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable --profile minimal
 
-# For now, we need to tell Debian we don't care that we're editing the system python
-# installation.
-# Some context in https://github.com/pypa/pip/issues/11381#issuecomment-1399263627
-RUN ${PYTHON_VERSION} -m pip install -q --no-cache-dir poetry==1.3.2 ${SYSTEM_PIP_INSTALL_SUFFIX}
+# Set the default Python version to the minimum supported by Synapse:
+# https://element-hq.github.io/synapse/latest/deprecation_policy.html
+ARG PYTHON_VERSION=3.10
+
+RUN --mount=type=bind,from=ghcr.io/astral-sh/uv:0.9.4,source=/uv,target=/bin/uv \
+        uv python install "$PYTHON_VERSION" && \
+        uv tool install [email protected]
+
+ENV PATH=/root/.local/bin:$PATH
 
 # As part of the Docker build, we attempt to pre-install Synapse's dependencies
 # in the hope that it speeds up the real install of Synapse. To make this work,
@@ -45,7 +47,7 @@ RUN mkdir /src
 
 # Download a cache of build dependencies to support offline mode.
 # These version numbers are arbitrary and were the latest at the time.
-RUN ${PYTHON_VERSION} -m pip download --dest /pypi-offline-cache \
+RUN "python${PYTHON_VERSION}" -m pip download --dest /pypi-offline-cache \
         poetry-core==1.1.0 setuptools==65.3.0 wheel==0.37.1 \
         setuptools-rust==1.5.1
 
@@ -57,7 +59,7 @@ RUN wget -q https://github.com/element-hq/synapse/archive/develop.tar.gz \
         tar -C /synapse --strip-components=1 -xf synapse.tar.gz && \
         ln -s -T /venv /synapse/.venv && \
         cd /synapse && \
-        poetry install -q --no-root --extras all && \
+        poetry install --no-root --extras all && \
         # Finally clean up the poetry cache and the copy of Synapse.
         # This must be done in the same RUN command, otherwise intermediate layers
         # of the Docker image will contain all the unwanted files we think we've