Problems with nexus as registry.mirror

[mpretzer]

Hi,

we're using a nexus-oss as registry mirror for docker hub. Using our existing yet old version of nexus works fine. However, podman fails to download images from a newer version of nexus, both using the same configuration.

My podman is:

Client:        Podman Engine
Version:       5.6.0
API Version:   5.6.0
Go Version:    go1.24.6
Git Commit:    da671ef6cfa3fc9ac6225c18f1dd0a70a951e43f
Built:         Fri Aug 15 00:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64

> cat .config/containers/registries.conf
[[registry]]
location = "docker.io"

[[registry.mirror]]
location = "entnexvpu0.xxx.de:8443"

This fails:

> podman --log-level=debug pull docker.io/library/alpine
[...]
time="2025-09-03T11:20:29+02:00" level=debug msg="GET https://entnexvpu0.xxx.de:8443/v2/"
time="2025-09-03T11:20:29+02:00" level=debug msg="Ping https://entnexvpu0.xxx.de:8443/v2/ status 401"
time="2025-09-03T11:20:29+02:00" level=debug msg="GET https://entnexvpu0.xxx.de:8443/v2/token?scope=repository%3Alibrary%2Falpine%3Apull&service=https%3A%2F%2Fentnexvpu0.xxx.de%3A8443%2Fv2%2Ftoken"
time="2025-09-03T11:20:29+02:00" level=debug msg="Increasing token expiration to: 60 seconds"
time="2025-09-03T11:20:29+02:00" level=debug msg="GET https://entnexvpu0.xxx.de:8443/v2/library/alpine/manifests/latest"
time="2025-09-03T11:20:29+02:00" level=debug msg="Content-Type from manifest GET is \"\""
time="2025-09-03T11:20:29+02:00" level=debug msg="Accessing \"entnexvpu0.xxx.de:8443/library/alpine:latest\" failed: reading manifest latest in entnexvpu0.xxx.de:8443/library/alpine: authentication required"
time="2025-09-03T11:20:29+02:00" level=debug msg="Trying to access \"docker.io/library/alpine:latest\""
[...]
[proceeds to download from docker.io]

Successfully using our old nexus (nexus oss 3.64.0-01): (hostname entnexvpc0 instead of entnexvpu0):

> podman --log-level=debug pull docker.io/library/alpine
[...]
time="2025-09-03T11:23:31+02:00" level=debug msg="GET https://entnexvpc0.xxx.de:8443/v2/"
time="2025-09-03T11:23:31+02:00" level=debug msg="Ping https://entnexvpc0.xxx.de:8443/v2/ status 401"
time="2025-09-03T11:23:31+02:00" level=debug msg="GET https://entnexvpc0.xxx.de:8443/v2/token?scope=repository%3Alibrary%2Falpine%3Apull&service=https%3A%2F%2Fentnexvpc0.xxx.de%3A8443%2Fv2%2Ftoken"
time="2025-09-03T11:23:31+02:00" level=debug msg="Increasing token expiration to: 60 seconds"
time="2025-09-03T11:23:31+02:00" level=debug msg="GET https://entnexvpc0.xxx.de:8443/v2/library/alpine/manifests/latest"
time="2025-09-03T11:23:31+02:00" level=debug msg="Content-Type from manifest GET is \"application/vnd.oci.image.index.v1+json\""
time="2025-09-03T11:23:31+02:00" level=debug msg="Using SQLite blob info cache at /home/matthias/.local/share/containers/cache/blob-info-cache-v1.sqlite"
time="2025-09-03T11:23:31+02:00" level=debug msg="Source is a manifest list; copying (only) instance sha256:eafc1edb577d2e9b458664a15f23ea1c370214193226069eb22921169fc7e43f for current system"
time="2025-09-03T11:23:31+02:00" level=debug msg="GET https://entnexvpc0.xxx.de:8443/v2/library/alpine/manifests/sha256:eafc1edb577d2e9b458664a15f23ea1c370214193226069eb22921169fc7e43f"
[...]

Using curl I compared the answers from both nexus instances and I cannot find a relevant difference (curl -sD - https://entnexvpc0.xxx.de:8443/v2/library/alpine/manifests/latest):

--- entnexvpc0  2025-09-03 11:36:01.800002090 +0200
+++ entnexvpu0  2025-09-03 11:35:54.807911313 +0200
@@ -1,16 +1,14 @@
 HTTP/1.1 200 OK
-Date: Wed, 03 Sep 2025 09:36:01 GMT
-Server: Nexus/3.64.0-01 (OSS)
-Strict-Transport-Security: max-age=31536000; includeSubDomains
+Date: Wed, 03 Sep 2025 09:35:54 GMT
+Server: Nexus/3.83.1-03 (COMMUNITY)
 X-Content-Type-Options: nosniff
-Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
+Content-Security-Policy: default-src https: data: blob: 'unsafe-inline'; script-src https: 'unsafe-inline' 'unsafe-eval'
+Strict-Transport-Security: max-age=31536000; includeSubDomains;
 X-XSS-Protection: 1; mode=block
 Docker-Distribution-Api-Version: registry/2.0
 ETag: "sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1"
-Last-Modified: Wed, 16 Jul 2025 10:40:27 GMT
+Last-Modified: Mon, 25 Aug 2025 13:19:51 GMT
 Docker-Content-Digest: sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
 Content-Type: application/vnd.oci.image.index.v1+json
 Content-Length: 9218
-Cache-Control: max-age=0
-Expires: Wed, 03 Sep 2025 09:36:01 GMT

I would greatly appreciate any hints.

[mpretzer]

Nevermind, nexus returns 401, I was mislead by time="2025-09-03T11:20:29+02:00" level=debug msg="Content-Type from manifest GET is \"\"".