Why am I experiencing DNS resolution failures only in the final stage of a multi-stage Docker Compose build?

[eljonny]

I am working on a custom deployment of some open source software where some of the code needs to be modified to provide a couple additional features that are not present, so I'm working on making modifications and rebuilding the main-app image.
I would ask this on the main repo (paperless-ngx) but I feel like this is more of a docker build issue than something specific to that project.
This is a project I'm working on in my spare time and I have been searching for a solution for this for the last 3 weeks.

The core problem is this:
When I run the docker multi-stage build, every stage but the final stage (called main-app) does not have DNS issues, but the final stage is not able to resolve deb.debian.org.

I cannot get Docker to build the main-app image on Linux (Xubuntu 24.04) but it works OK in Docker Desktop on Windows, which is bizarre and usually the case is the opposite, works right on Linux and not on Windows.

Docker versions:

• Docker: Docker version 28.2.2, build 28.2.2-0ubuntu1~24.04.1
• Docker Compose: Docker Compose version 2.37.1+ds1-0ubuntu2~24.04.1
• Docker Buildx: github.com/docker/buildx 0.21.3 0.21.3-0ubuntu1~24.04.1

Here is the Dockerfile (unmodified from the 2.17.1 source):

# syntax=docker/dockerfile:1
# https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md

# Stage: compile-frontend
# Purpose: Compiles the frontend
# Notes:
#  - Does PNPM stuff with Typescript and such
FROM --platform=$BUILDPLATFORM docker.io/node:20-bookworm-slim AS compile-frontend

COPY ./src-ui /src/src-ui

WORKDIR /src/src-ui
RUN set -eux \
  && npm update -g pnpm \
  && npm install -g corepack@latest \
  && corepack enable \
  && pnpm install

ARG PNGX_TAG_VERSION=
# Add the tag to the environment file if its a tagged dev build
RUN set -eux && \
case "${PNGX_TAG_VERSION}" in \
  dev|beta|fix*|feature*) \
    sed -i -E "s/tag: '([a-z\.]+)'/tag: '${PNGX_TAG_VERSION}'/g" /src/src-ui/src/environments/environment.prod.ts \
    ;; \
esac

RUN set -eux \
  && ./node_modules/.bin/ng build --configuration production

# Stage: s6-overlay-base
# Purpose: Installs s6-overlay and rootfs
# Comments:
#  - Don't leave anything extra in here either
FROM ghcr.io/astral-sh/uv:0.7.9-python3.12-bookworm-slim AS s6-overlay-base

WORKDIR /usr/src/s6

# https://github.com/just-containers/s6-overlay#customizing-s6-overlay-behaviour
ENV \
    S6_BEHAVIOUR_IF_STAGE2_FAILS=2 \
    S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 \
    S6_VERBOSITY=1 \
    PATH=/command:$PATH

# Buildx provided, must be defined to use though
ARG TARGETARCH
ARG TARGETVARIANT
# Lock this version
ARG S6_OVERLAY_VERSION=3.2.1.0

ARG S6_BUILD_TIME_PKGS="curl \
                        xz-utils"

RUN set -eux \
    && echo "Installing build time packages" \
      && apt-get update \
      && apt-get install --yes --quiet --no-install-recommends ${S6_BUILD_TIME_PKGS} \
    && echo "Determining arch" \
      && S6_ARCH="" \
      && if [ "${TARGETARCH}${TARGETVARIANT}" = "amd64" ]; then S6_ARCH="x86_64"; \
      elif [ "${TARGETARCH}${TARGETVARIANT}" = "arm64" ]; then S6_ARCH="aarch64"; fi\
      && if [ -z "${S6_ARCH}" ]; then { echo "Error: Not able to determine arch"; exit 1; }; fi \
    && echo "Installing s6-overlay for ${S6_ARCH}" \
      && curl --fail --silent --no-progress-meter --show-error --location --remote-name-all --parallel --parallel-max 4 \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz.sha256" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_ARCH}.tar.xz" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_ARCH}.tar.xz.sha256" \
      && echo "Validating s6-archive checksums" \
        && sha256sum --check ./*.sha256 \
      && echo "Unpacking archives" \
        && tar --directory / -Jxpf s6-overlay-noarch.tar.xz \
        && tar --directory / -Jxpf s6-overlay-${S6_ARCH}.tar.xz \
      && echo "Removing downloaded archives" \
        && rm ./*.tar.xz \
        && rm ./*.sha256 \
    && echo "Cleaning up image" \
      && apt-get --yes purge ${S6_BUILD_TIME_PKGS} \
      && apt-get --yes autoremove --purge \
      && rm -rf /var/lib/apt/lists/*

# Copy our service defs and filesystem
COPY ./docker/rootfs /

# Stage: main-app
# Purpose: The final image
# Comments:
#  - Don't leave anything extra in here
FROM s6-overlay-base AS main-app

LABEL org.opencontainers.image.authors="paperless-ngx team <[email protected]>"
LABEL org.opencontainers.image.documentation="https://docs.paperless-ngx.com/"
LABEL org.opencontainers.image.source="https://github.com/paperless-ngx/paperless-ngx"
LABEL org.opencontainers.image.url="https://github.com/paperless-ngx/paperless-ngx"
LABEL org.opencontainers.image.licenses="GPL-3.0-only"

ARG DEBIAN_FRONTEND=noninteractive

# Buildx provided, must be defined to use though
ARG TARGETARCH

# Can be workflow provided, defaults set for manual building
ARG JBIG2ENC_VERSION=0.30
ARG QPDF_VERSION=11.9.0
ARG GS_VERSION=10.03.1

# Set Python environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    # Ignore warning from Whitenoise about async iterators
    PYTHONWARNINGS="ignore:::django.http.response:517" \
    PNGX_CONTAINERIZED=1 \
    # https://docs.astral.sh/uv/reference/settings/#link-mode
    UV_LINK_MODE=copy \
    UV_CACHE_DIR=/cache/uv/

#
# Begin installation and configuration
# Order the steps below from least often changed to most
#

# Packages need for running
ARG RUNTIME_PACKAGES="\
  # General utils
  curl \
  # Docker specific
  gosu \
  # Timezones support
  tzdata \
  # fonts for text file thumbnail generation
  fonts-liberation \
  gettext \
  ghostscript \
  gnupg \
  icc-profiles-free \
  imagemagick \
  # PostgreSQL
  postgresql-client \
  # MySQL / MariaDB
  mariadb-client \
  # OCRmyPDF dependencies
  tesseract-ocr \
  tesseract-ocr-eng \
  tesseract-ocr-deu \
  tesseract-ocr-fra \
  tesseract-ocr-ita \
  tesseract-ocr-spa \
  unpaper \
  pngquant \
  jbig2dec \
  # lxml
  libxml2 \
  libxslt1.1 \
  # itself
  qpdf \
  # Mime type detection
  file \
  libmagic1 \
  media-types \
  zlib1g \
  # Barcode splitter
  libzbar0 \
  poppler-utils"

# Install basic runtime packages.
# These change very infrequently
RUN set -eux \
  echo "Installing system packages" \
    && apt-get update \
    && apt-get install --yes --quiet --no-install-recommends ${RUNTIME_PACKAGES} \
    && echo "Installing pre-built updates" \
      && curl --fail --silent --no-progress-meter --show-error --location --remote-name-all --parallel --parallel-max 4 \
        https://github.com/paperless-ngx/builder/releases/download/qpdf-${QPDF_VERSION}/libqpdf29_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/qpdf-${QPDF_VERSION}/qpdf_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/libgs10_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/ghostscript_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/libgs10-common_${GS_VERSION}.dfsg-1_all.deb \
        https://github.com/paperless-ngx/builder/releases/download/jbig2enc-${JBIG2ENC_VERSION}/jbig2enc_${JBIG2ENC_VERSION}-1_${TARGETARCH}.deb \
      && echo "Installing qpdf ${QPDF_VERSION}" \
        && dpkg --install ./libqpdf29_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        && dpkg --install ./qpdf_${QPDF_VERSION}-1_${TARGETARCH}.deb \
      && echo "Installing Ghostscript ${GS_VERSION}" \
        && dpkg --install ./libgs10-common_${GS_VERSION}.dfsg-1_all.deb \
        && dpkg --install ./libgs10_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        && dpkg --install ./ghostscript_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
      && echo "Installing jbig2enc" \
        && dpkg --install ./jbig2enc_${JBIG2ENC_VERSION}-1_${TARGETARCH}.deb \
      && echo "Configuring imagemagick" \
        && cp /etc/ImageMagick-6/paperless-policy.xml /etc/ImageMagick-6/policy.xml \
      && echo "Cleaning up image layer" \
        && rm --force --verbose *.deb \
    && rm --recursive --force --verbose /var/lib/apt/lists/*

WORKDIR /usr/src/paperless/src/

# Python dependencies
# Change pretty frequently
COPY --chown=1000:1000 ["pyproject.toml", "uv.lock", "/usr/src/paperless/src/"]

# Packages needed only for building a few quick Python
# dependencies
ARG BUILD_PACKAGES="\
  build-essential \
  # https://github.com/PyMySQL/mysqlclient#linux
  default-libmysqlclient-dev \
  pkg-config"

# hadolint ignore=DL3042
RUN --mount=type=cache,target=${UV_CACHE_DIR},id=python-cache \
  set -eux \
  && echo "Installing build system packages" \
    && apt-get update \
    && apt-get install --yes --quiet --no-install-recommends ${BUILD_PACKAGES} \
  && echo "Installing Python requirements" \
    && uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt \
    && uv pip install --system --no-python-downloads --python-preference system --requirements requirements.txt \
  && echo "Installing NLTK data" \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" snowball_data \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" stopwords \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" punkt_tab \
  && echo "Cleaning up image" \
    && apt-get --yes purge ${BUILD_PACKAGES} \
    && apt-get --yes autoremove --purge \
    && apt-get clean --yes \
    && rm --recursive --force --verbose *.whl \
    && rm --recursive --force --verbose /var/lib/apt/lists/* \
    && rm --recursive --force --verbose /tmp/* \
    && rm --recursive --force --verbose /var/tmp/* \
    && rm --recursive --force --verbose /var/cache/apt/archives/* \
    && truncate --size 0 /var/log/*log

# copy backend
COPY --chown=1000:1000 ./src ./

# copy frontend
COPY --from=compile-frontend --chown=1000:1000 /src/src/documents/static/frontend/ ./documents/static/frontend/

# add users, setup scripts
# Mount the compiled frontend to expected location
RUN set -eux \
  && sed -i '1s|^#!/usr/bin/env python3|#!/command/with-contenv python3|' manage.py \
  && echo "Setting up user/group" \
    && addgroup --gid 1000 paperless \
    && useradd --uid 1000 --gid paperless --home-dir /usr/src/paperless paperless \
  && echo "Creating volume directories" \
    && mkdir --parents --verbose /usr/src/paperless/data \
    && mkdir --parents --verbose /usr/src/paperless/media \
    && mkdir --parents --verbose /usr/src/paperless/consume \
    && mkdir --parents --verbose /usr/src/paperless/export \
  && echo "Creating gnupg directory" \
    && mkdir -m700 --verbose /usr/src/paperless/.gnupg \
  && echo "Adjusting all permissions" \
    && chown --from root:root --changes --recursive paperless:paperless /usr/src/paperless \
  && echo "Collecting static files" \
    && s6-setuidgid paperless python3 manage.py collectstatic --clear --no-input --link \
    && s6-setuidgid paperless python3 manage.py compilemessages

VOLUME ["/usr/src/paperless/data", \
        "/usr/src/paperless/media", \
        "/usr/src/paperless/consume", \
        "/usr/src/paperless/export"]

ENTRYPOINT ["/init"]

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=10s --retries=5 CMD [ "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000" ]

I have tried the following things:

• Disabled AppArmor at user level with the following command: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
• Disabled AppArmor at the kernel level, via grub using this script
• Configured the docker daemon with DNS and private network settings (change the network with the "bip" key)
• Configured the docker daemon with only DNS settings (see daemon config below)
• Created a new builder with the following command to make sure everything is using the docker bridge network: docker buildx create --platform 'linux/amd64' --use --name 'pngx-builder' --driver 'docker-container' --driver-opt 'network=bridge' --buildkitd-flags '--oci-worker-net bridge' --bootstrap
• Created another new builder with the following command to use direct host networking: docker buildx create --platform 'linux/amd64' --use --name 'pngx-builder-hn' --driver 'docker-container' --driver-opt 'network=host' --buildkitd-flags '--allow-insecure-entitlement network.host --oci-worker-net host' --bootstrap
• Run the build with --no-cache and cached mode
• Run the build with the basic command: docker compose build
• Run the build with a more complete command, that uses the non-default builder, to try to help debug and find the root cause of the issue: docker compose --all-resources --progress=plain build --no-cache --builder 'pngx-builder' --with-dependencies
• Run the build with a more complete command, that uses the non-default builder with direct host networking: docker compose --all-resources --progress=plain build --no-cache --builder 'pngx-builder-hn' --with-dependencies
• Run the more complete command on the default builder
• Run the build by itself outside docker compose
• Configure systemd-resolved to create an intermediate DNS resolution server between docker and host network within the private docker network
• Configure systemd-resolved with a set of known-good DNS servers (Cloudflare, Quad9, OpenDNS, Google, my local router which routes to Spectrum DNS)
• Configure systemd-resolved by disabling the stub resolver at 127.0.0.53 so everything just gets routed the way that it would normally without systemd-resolved
• Run docker buildx prune --all --force repeatedly
• Disable Docker Buildx and use the classic builder
  • It seems like DNS might work here but the build is not compatible for other reasons that I don't completely understand
  • I have to modify the Dockerfile here because the classic builder doesn't support multi-platform builds
• Restarting docker daemon (so many times)
• Ran cat /etc/resolv.conf in the Dockerfile RUN command in the main-app image and verified the name servers are present as they should be.
• Confirmed that UFW is disabled with command: sudo ufw status

This is my current daemon config (at /etc/docker/daemon.json):

{
    "dns": [
        "1.1.1.1",
        "1.0.0.1",
        "9.9.9.9",
        "208.67.222.222",
        "208.67.220.220"
    ]
}

This is the current configuration for systemd-resolved that disables the stub resolver (at /etc/systemd/resolved.conf.d/docker.conf):

[Resolve]
DNSStubListener=no

This is the current configuration for ensuring known-good DNS name servers are available through systemd-resolved (at /etc/systemd/resolved.conf.d/userdns.conf):

[Resolve]
#Cloudflare, Quad9, and OpenDNS
DNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001 9.9.9.9 149.112.112.112 2620:fe::fe 2620:fe::9 208.67.222.222 208.67.220.220
#Google, Spectrum
FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844 192.168.1.1 2600:6c4e:4400:1950::1

Something is blocking DNS resolution in the final image stage in the multi-stage build (main-app), and I cannot for the life of me figure out what it is.
This is what happens for the big RUN command in main-app:

#17 8.967 W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'deb.debian.org'
#17 8.968 W: Failed to fetch http://deb.debian.org/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'deb.debian.org'
#17 8.968 W: Failed to fetch http://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Temporary failure resolving 'deb.debian.org'

I have seen many mentions of issues like this on the Docker forums and in limited instances elsewhere, but nowhere have I found a concrete resolution for it.

I feel like I have tried everything, I hope someone here might be able to help. Here is the full build log.

Also posted/asked on SO.