`latest` tag for `jenkins/inbound-agent` seems to flip between Debian "trixie" and "bookworm" as the OS (AMD)

[aRobinson-R7]

Jenkins and plugins versions report

Occurs during building of image, rather than executing Jenkins

• using jenkins/inbound-agent:latest
• attempting to pin the version of docker installed

What Operating System are you using (both controller, and any agents involved in the problem)?

Debian AMD

Reproduction steps

1. Use the jenkins/inbound-agent:latest image to build your own Docker image
2. Purposefully install a software version pinned to trixie e.g apt-get -y install docker-ce=5:28.1.0-1~debian.13~trixie
3. Builds will occasionally fail as the image pulled is bookworm (based on release cadence of new images, run this several times around when new images come out)

Example set of commands to install Docker that can fail

RUN install -m 0755 -d /etc/apt/keyrings && \
  curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
  chmod a+r /etc/apt/keyrings/docker.gpg && \
  echo "deb [arch=\"$(dpkg --print-architecture)\" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
  cat /etc/apt/sources.list.d/docker.list && \
  apt-get update && \
  apt-get -y install docker-ce=5:28.1.0-1~debian.13~trixie skopeo && \
  usermod -aG docker jenkins && \
  docker --help > /dev/null

Expected Results

when using latest image tag, trixie will always be pulled, image builds will not fail

Actual Results

Builds occasionally fail with the following error:

17:21:48   > [jenkins-agent-default linux/amd64 base  6/16] 
17:21:48  1.860 Reading package lists...
17:21:48  2.262 Building dependency tree...
17:21:48  2.378 Reading state information...
17:21:48  2.410 Package docker-ce is not available, but is referred to by another package.
17:21:48  2.410 This may mean that the package is missing, has been obsoleted, or
17:21:48  2.410 is only available from another source
17:21:48  2.410 However the following packages replace it:
17:21:48  2.410   docker-ce-cli
17:21:48  2.410 
17:21:48  2.414 E: Version '5:28.1.0-1~debian.13~trixie' for 'docker-ce' was not found

The following part of the command makes it clear it's bookworm:

echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

Anything else?

Sounds a successor to or related to #1059

This appears to occur around the time new images are released, as this again happened to me when attempting to debug this locally this week.
When the bookworm image is released, it gets tagged as latest
Then when the trixie image is released, it then gets tagged as latest

Presume there is just an additional tag in the bookworm configuration
I have no evidence of this occurring before this month (Jan) as we do not build the image on a regular cadence

Are you interested in contributing a fix?

No response

(edited for back tick issues)

[MarkEWaite]

I suspect you have some local caching issue rather than an issue with the Jenkins agent container images. We have not built a container image based on bookworm in a long time. The last controller image to provide Bookworm was 2.529 as noted in the 2.530 changelog in Sep 2025. The last inbound agent to provide Bookworm was 3341.v0766d82b_dec0-2 Sep 30, 2025.

[aRobinson-R7]

Interesting, thank you for the response @MarkEWaite - I'd originally thought that based on the previous issue I linked.

However if you look on Dockerhub, bookworm images were published 9 days ago for several JDKs (with latest versions), which could be the cause of this (as potentially trixie overwrote the bad tag)

I do not know the release cadence of the Jenkins images, so do not know this could be linked

[aRobinson-R7]

As a note we've only been pinning the docker version since the end of November, these builds had only started failing at some time this month and we've now moved to latest-trixie to get around it.

I suspect you have some local caching issue rather than an issue

Had occurred on my local machine while debugging this as well as in AWS where we normally build these things (pulling images from Dockerhub)

As I said it looks to have been resolved with the latest trixie images

[dduportal]

Interesting, let me check the build history in our private Jenkins controller. I have a feeling we still have these "ghost" build which are triggered while they should not :'(

[dduportal]

In the mean time @aRobinson-R7 could you share the day of a failing build? Your build log mentions a time 17:21:48: is it UTC? CET? Other?

=> It would help us to filter the build history

[dduportal]

However if you look on Dockerhub, bookworm images were published 9 days ago for several JDKs (with latest versions), which could be the cause of this (as potentially trixie overwrote the bad tag)

I missed this comment. That helps, thanks

[dduportal]

I confirm it is, again, a "ghost" build on trusted.ci: https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/2/ and https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/3/.

Ping @daniel-beck @MarkEWaite @timja we'll need your help here to analyse. There are no valid reason why this build would start:

• Only tags are discovered in this MB pipeline job
• And it also have a build strategy to NOT build any tags older than 3 days old
  • This is working: when we publish a non annotated tags on a commit older than 3 days, it does not start the build (we've had this during security advisories many times)
• Builds says "Branch Indexing" which should not even start a build since there is already a successful build
  • As if the "previous build" state is not visible in some cases 🤔
• The only "weird" thing is the "Config changed since last build" on both "ghost" builds.

As if the config history was taking over the build triggers and policy

[dduportal]

@aRobinson-R7 I suggest that you update your setup to avoid "latest"-like labels in any case: using the strict tags would avoid such bad surprises: jenkins/inbound-agent:3355.v388858a_47b_33-7-jdk17 or jenkins/inbound-agent:3355.v388858a_47b_33-7-jdk21 for instance.

[aRobinson-R7]

Thanks for looking @dduportal

[aRobinson-R7]

In the mean time @aRobinson-R7 could you share the day of a failing build? Your build log mentions a time 17:21:48: is it UTC? CET? Other?

The failing build I linked was from the 19th January, I believe the time is GMT

[lemeurherve]

Recent threads in #jenkinsci/release gitter channel about the same issue of old controller tags republished:

• https://matrix.to/#/!JlkqzpdEnsUUuVtjgE:matrix.org/$Umz_fnStdhmzhPD4RGasDXt8L_nIHD4jiB8wxCR-9Qg?via=matrix.org&via=gitter.im
• https://matrix.to/#/!JlkqzpdEnsUUuVtjgE:matrix.org/$QZSgqMLivLeiP2cTO9oJJScxH7HIiueQS7mNZs7HobU?via=matrix.org&via=gitter.im
• https://matrix.to/#/!JlkqzpdEnsUUuVtjgE:matrix.org/$kVbKyFeSIgW167dJskMNmsZ5CSPKMJ9UrEnn7B9nPIg?via=matrix.org&via=gitter.im

The docker controller publication job has been temporary disabled on trusted.ci.jenkins.io to prevents those unexpected republications.

I'm planning to migrate them to release.ci.jenkins.io to hopefully stabilize them (but not only), more to come in the incoming days.

[lemeurherve]

I confirm it is, again, a "ghost" build on trusted.ci: https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/2/* and https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/3/* .

Unfortunately those builds don't exist anymore (?)

FWIW, last success dates from docker agents job on trusted.ci.jenkins.io LGTM:

[dduportal]

I confirm it is, again, a "ghost" build on trusted.ci: https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/2/* and https://trusted.ci.jenkins.io/job/Containers/job/Agents/job/docker-agents/view/tags/job/3299.v0d0d06908537-1/3/* .

Unfortunately those builds don't exist anymore (?)

That is really weird and unexpected 🤔