GESIS server fails when pushing the image

[rgaiacs]

I tested the GESIS server with the "empty" Git repository https://github.com/rgaiacs/mwe4mybinder.org and the user log shows

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2docker7j5zbjeh'...
HEAD is now at f1c2b5a Initial commit
Python version unspecified, using current default Python version 3.10. This will change in the future.Building conda environment for python=3.10
Using PythonBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 4.85kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 DONE 0.6s

#3 [ 1/32] FROM docker.io/library/buildpack-deps:jammy@sha256:1a63cc50ec6f4f45440121af59c47f78ca390607714ac0886c1588d118526b40
#3 DONE 0.0s

#4 [13/18] RUN chown jovyan:jovyan /home/jovyan
#4 CACHED

#5 [internal] load .dockerignore
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 55.42kB 0.0s done
#6 DONE 0.0s

#7 [11/16] RUN if [ ! -d "/home/jovyan" ]; then         /usr/bin/install -o jovyan -g jovyan -d "/home/jovyan";     fi
#7 CACHED

#8 [ 7/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e10-2dlinux-2d64-2elock-2bb6e8 /tmp/env/environment.lock
#8 CACHED

#9 [ 3/32] RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
#9 CACHED

#10 [ 9/16] RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
#10 CACHED

#11 [13/16] RUN chown jovyan:jovyan /home/jovyan
#11 CACHED

#12 [ 4/32] RUN groupadd         --gid 1000         jovyan &&     useradd         --comment "Default user"         --create-home         --gid 1000         --no-log-init         --shell /bin/bash         --uid 1000         jovyan
#12 CACHED

#13 [ 5/32] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        gettext-base        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#13 CACHED

#14 [12/16] WORKDIR /home/jovyan
#14 CACHED

#15 [ 8/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-637204 /tmp/install-base-env.bash
#15 CACHED

#16 [ 2/32] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#16 CACHED

#17 [10/16] RUN mkdir -p /srv/npm && chown -R jovyan:jovyan /srv/npm
#17 CACHED

#18 [ 6/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e67d51 /etc/profile.d/activate-conda.sh
#18 CACHED

#19 [14/16] COPY --chown=1000:1000 src/ /home/jovyan/
#19 DONE 0.0s

#20 [15/16] COPY /python3-login /usr/local/bin/python3-login
#20 DONE 0.0s

#21 [16/16] COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
#21 DONE 0.1s

#22 exporting to image
#22 exporting layers 0.0s done
#22 writing image sha256:cf74014e1a659ce70af34eeb807f439089263ff61a87ab84dd71fadf9f7aee71 done
#22 naming to docker.io/gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d done
#22 DONE 0.0s

#23 pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d with docker
#23 pushing layer 3fe195e5c769
#23 pushing layer 7416ce608053
#23 pushing layer 2d85a96be1ce
#23 pushing layer ae098c20d483
#23 pushing layer 5f70bf18a086
#23 pushing layer 45cf5b5745c2
#23 pushing layer 7e5d8dc46f59
#23 pushing layer cdc321148327
#23 pushing layer f57802063fea
#23 pushing layer 468f61133dd2
#23 pushing layer d3e42451921b
#23 pushing layer cff0c8b3a131
#23 pushing layer 0a72ce9f0ac6
#23 pushing layer 330a3443e078
#23 pushing layer a3e1a9f8dd1f
#23 pushing layer 3088d1636682
#23 pushing layer b7d4ab99ea66
#23 pushing layer 270a1170e7e3
#23 pushing layer 3fe195e5c769 0.9s done
#23 pushing layer 7416ce608053 0.9s done
#23 pushing layer 2d85a96be1ce 0.9s done
#23 pushing layer ae098c20d483 0.9s done
#23 pushing layer 5f70bf18a086 0.9s done
#23 pushing layer 45cf5b5745c2 0.9s done
#23 pushing layer 7e5d8dc46f59 0.9s done
#23 pushing layer cdc321148327 0.9s done
#23 pushing layer f57802063fea 0.9s done
#23 pushing layer 468f61133dd2 0.9s done
#23 pushing layer d3e42451921b 0.9s done
#23 pushing layer cff0c8b3a131 0.9s done
#23 pushing layer 0a72ce9f0ac6 0.9s done
#23 pushing layer 330a3443e078 0.9s done
#23 pushing layer a3e1a9f8dd1f 0.9s done
#23 pushing layer 3088d1636682 0.9s done
#23 pushing layer b7d4ab99ea66 0.9s done
#23 pushing layer 270a1170e7e3 0.9s done
#23 ERROR: denied: requested access to the resource is denied
------
 > pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d with docker:
------
ERROR: denied: requested access to the resource is denied
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:f1c2b5a658c819212139ef8b82ad335055f6c41d', '--platform', 'linux/amd64', '/tmp/tmpzmpw3wvj']' returned non-zero exit status 1.

This is happening after #3225 when mybinder.org started using docker buildx. My understanding of the error is that the credentials to the container registry used by GESIS are incorrect. But they were not changed.

@yuvipanda could you point to me where to configure the credentials used by docker buildx? Thanks!

cc @arnim

[yuvipanda]

@rgaiacs if you look under secrets/config/hetzner-2i2c.yaml, you'll see them under binderhub.registry and binderhub.jupyterhub.imagePullSecret

[yuvipanda]

Temporarily taken #3254 out of rotation while we fix this. I hadn't fully realized that the CI/CD tests won't catch GESIS failures on this repo. Sorry @rgaiacs

[yuvipanda]

i was also looking at grafana, and didn't spot the gesis failures there. will need to look into that as well

[rgaiacs]

you'll see them under binderhub.registry

No change on GESIS configuration. We have

binderhub:
  registry:
    username: ***
    password: ***

I looked at the Docker Hub account that GESIS is using and the last time that we was able to push an image was 5 days ago (6 Mar 2025 at 7:46 pm UTC+1).

On 6 Mar 2025, we updated the GESIS deployment to sync the KubernetesBuildExecutor.build_image. No other changes.

@yuvipanda do you know if buildx is configured to work only with access tokens instead of password?

Another possible cause for this problem is the changes announced at Revisiting Docker Hub Policies: Prioritizing Developer Experience.

Temporarily taken #3254 out of rotation

👍 Good call.

was also looking at grafana, and didn't spot the gesis failures there.

It is hard to spot because it only fails when building a new image. GESIS is operating a "perpetual" container register on Docker Hub and we pull existing images.

[rgaiacs]

I tested a local build and push from my laptop using the credentials from GESIS and it worked. The Dockerfile was

FROM alpine:3.21.3

WORKDIR /mnt/test

COPY ./README.md /mnt/test/README.md

and running

docker buildx build --progress plain --push --tag gesiscss/mwe-3253:1 .

returned

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 112B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:3.21.3
#2 ...

#3 [auth] library/alpine:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:3.21.3
#2 DONE 1.2s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/3] FROM docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c
#5 resolve docker.io/library/alpine:3.21.3@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c 0.0s done
#5 CACHED

#6 [internal] load build context
#6 transferring context: 37B done
#6 DONE 0.0s

#7 [2/3] WORKDIR /mnt/test
#7 DONE 0.0s

#8 [3/3] COPY ./README.md /mnt/test/README.md
#8 DONE 0.0s

#9 exporting to image
#9 exporting layers 0.1s done
#9 exporting manifest sha256:d99bff2a38eded1ec1ffa8116720d0bb01e9b73c06c10a8fd910841bded2319c
#9 exporting manifest sha256:d99bff2a38eded1ec1ffa8116720d0bb01e9b73c06c10a8fd910841bded2319c 0.0s done
#9 exporting config sha256:bd8f7902d13d3061b30f43d7129136c262185d2d35e1b5b94f5bf05c4239ba9d 0.0s done
#9 exporting attestation manifest sha256:91ec0c09a52322d3e56c835022f7792c49ab77cc876f87fa66918e18f7be0fc2 0.0s done
#9 exporting manifest list sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a 0.0s done
#9 naming to docker.io/gesiscss/mwe-3253:1 done
#9 unpacking to docker.io/gesiscss/mwe-3253:1 0.0s done
#9 pushing layers
#9 ...

#10 [auth] gesiscss/mwe-3253:pull,push token for registry-1.docker.io
#10 DONE 0.0s

#9 exporting to image
#9 pushing layers 7.4s done
#9 pushing manifest for docker.io/gesiscss/mwe-3253:1@sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a
#9 pushing manifest for docker.io/gesiscss/mwe-3253:1@sha256:6e7d45a28a8f7a82d0799a617c2f825ceadb7cd141a249726bef07f2a339386a 1.7s done
#9 DONE 9.8s

#11 pushing gesiscss/mwe-3253:1 with docker
#11 pushing layer 108297fb44bf
#11 pushing layer dc672d261718 0.1s
#11 pushing layer f18232174bc9 0.1s
#11 pushing layer f1b6a732c212 0.1s
#11 pushing layer 108297fb44bf 2.9s done
#11 pushing layer dc672d261718 2.9s done
#11 pushing layer f18232174bc9 2.9s done
#11 pushing layer f1b6a732c212 2.9s done
#11 DONE 3.2s

But I tested from the GESIS server and I got the same error.

#22 pushing layer 270a1170e7e3 0.7s done
#22 ERROR: denied: requested access to the resource is denied
------
 > pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:6e8f31e93fd476581c07bc6a8bf407176033424d with docker:
------
ERROR: denied: requested access to the resource is denied
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:6e8f31e93fd476581c07bc6a8bf407176033424d', '--platform', 'linux/amd64', '/tmp/tmpmzekq582']' returned non-zero exit status 1.Error in event stream: Error

@yuvipanda do you know a easy way to start a debugging build pod on Kubernetes with the same configuration?

[rgaiacs]

The good news is that I was able to replicate the error locally.

I used the dirty trick to provide a postBuild with

#!/bin/bash

tail -f /dev/null

This will keep the build pod on the server running and allowing me to inspect and debug it.

I copied the config.json provided by the secret build-docker-config to another machine (bare metal Linux) and run

docker buildx build --progress plain --push --tag gesiscss/mwe-3253:1 .

This time, I got the error

ERROR: denied: requested access to the resource is denied

After some reading and ChatGPT prompt, my understanding is that Docker Hub finally deprecated authentication with base64. On 14 August 2024, Docker announced Deprecation of Password Logins on CLI with Docker SSO Enforcement. But they didn't change the settings for other users at that time. The docker cli reference manual includes the following section:

Provide a password using STDIN (--password-stdin)

To run the docker login command non-interactively, you can set the --password-stdin flag to provide a password through STDIN. Using STDIN prevents the password from ending up in the shell's history, or log-files.

The following example reads a password from a file, and passes it to the docker login command using STDIN:

cat ~/my_password.txt | docker login --username foo --password-stdin

This will require changes on BinderHub.

[rgaiacs]

Please follow up jupyterhub/binderhub#1943.

[rgaiacs]

The problem is caused by a missing trailing slash in config.json, see jupyterhub/binderhub#1945 for a fix.

I'm trying to find if this change is on docker client.

From	Package	Docker version	build	docker login generates config.json with trailing slash
		28.0.1	068a01e	✔
Ubuntu 22.04.4 LTS	snap	27.5.1	a187fa5	✔
mybinder.org		27.3.1	ce1223035ac3ab8922717092e63a184cf67b493d	✔
Debian 12	docker.io	20.10.24	297e128	✔

[yuvipanda]

We now do use docker login with --password-stdin for login: https://github.com/jupyterhub/repo2docker/blob/main/repo2docker/docker.py#L200. That article seems to imply this is only true for accounts that enforce SSO - is the gesis org enforcing SSO? I tried this locally with my yuvipanda account and it seems to work fine.

fwiw, this is also why I avoid using any LLM tools while debugging - they often lead me down wrong paths but they don't really have any way to know it's wrong...

[rgaiacs]

We now do use docker login with --password-stdin for login

Awesome.

But my understanding is that BinderHub is creating /root/.docker/config.json using a ConfigMap defined in https://github.com/jupyterhub/binderhub/blob/c07d252a98f7906ec36d3c152e60f7bbd2ae3319/helm-chart/binderhub/templates/secret.yaml#L34-L42 and run docker login results in the following error

Error saving credentials: rename /root/.docker/config.json1199537460 /root/.docker/config.json: device or resource busy

I tried to discover if this was caused by change on the docker client but I couldn't install a wide range of versions.

is the gesis org enforcing SSO?

Not at the moment. It works OK for GESIS credentials when https://index.docker.io/v1/ has the trailing slash.

[rgaiacs]

I confirm that after #3258 (integrated to GESIS code tree in gesiscss/orc2@2c6c90e) the secret binder-build-docker-config now has the trailing slash, i.e.

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "***"
    }
  }
}

I also tested launching a minimal working example that requires the image container creation on the GESIS server and the push of the image was successful.

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2docker8migkrov'...
HEAD is now at 08de52c Update README.md
Python version unspecified, using current default Python version 3.10. This will change in the future.Building conda environment for python=3.10
Using PythonBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 4.85kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 ...

#3 [auth] library/buildpack-deps:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/library/buildpack-deps:jammy
#2 DONE 1.2s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [ 1/16] FROM docker.io/library/buildpack-deps:jammy@sha256:1a63cc50ec6f4f45440121af59c47f78ca390607714ac0886c1588d118526b40
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 60.34kB 0.0s done
#6 DONE 0.0s

#7 [ 3/16] RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
#7 CACHED

#8 [12/16] WORKDIR /home/jovyan
#8 CACHED

#9 [ 2/16] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#9 CACHED

#10 [ 6/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e67d51 /etc/profile.d/activate-conda.sh
#10 CACHED

#11 [ 8/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-637204 /tmp/install-base-env.bash
#11 CACHED

#12 [ 9/16] RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-base-env.bash' && rm -rf /tmp/install-base-env.bash /tmp/env
#12 CACHED

#13 [ 4/16] RUN groupadd         --gid 1000         jovyan &&     useradd         --comment "Default user"         --create-home         --gid 1000         --no-log-init         --shell /bin/bash         --uid 1000         jovyan
#13 CACHED

#14 [10/16] RUN mkdir -p /srv/npm && chown -R jovyan:jovyan /srv/npm
#14 CACHED

#15 [11/16] RUN if [ ! -d "/home/jovyan" ]; then         /usr/bin/install -o jovyan -g jovyan -d "/home/jovyan";     fi
#15 CACHED

#16 [ 5/16] RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        gettext-base        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
#16 CACHED

#17 [ 7/16] COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e12-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e10-2dlinux-2d64-2elock-2bb6e8 /tmp/env/environment.lock
#17 CACHED

#18 [13/16] RUN chown jovyan:jovyan /home/jovyan
#18 CACHED

#19 [14/16] COPY --chown=1000:1000 src/ /home/jovyan/
#19 DONE 0.0s

#20 [15/16] COPY /python3-login /usr/local/bin/python3-login
#20 DONE 0.0s

#21 [16/16] COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
#21 DONE 0.0s

#22 exporting to image
#22 exporting layers 0.0s done
#22 writing image sha256:5d1518fda1ab9e55f1d78d694060d2492a6794a06eb557d0a0d69e2973146f4d done
#22 naming to docker.io/gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:08de52c3baf12b50614bd0bdd104c6fce5f7915f done
#22 DONE 0.0s

#23 pushing gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:08de52c3baf12b50614bd0bdd104c6fce5f7915f with docker
#23 pushing layer 233df6498cd6
#23 pushing layer e33ffcf7d290
#23 pushing layer 736f4a42301c
#23 pushing layer ae098c20d483
#23 pushing layer 5f70bf18a086
#23 pushing layer 45cf5b5745c2
#23 pushing layer 7e5d8dc46f59
#23 pushing layer cdc321148327
#23 pushing layer f57802063fea
#23 pushing layer 468f61133dd2
#23 pushing layer d3e42451921b
#23 pushing layer cff0c8b3a131
#23 pushing layer 0a72ce9f0ac6
#23 pushing layer 330a3443e078
#23 pushing layer a3e1a9f8dd1f
#23 pushing layer 3088d1636682
#23 pushing layer b7d4ab99ea66
#23 pushing layer 270a1170e7e3
#23 pushing layer e33ffcf7d290 512B / 374B 0.8s
#23 pushing layer 233df6498cd6 6.66kB / 3.36kB 0.9s
#23 pushing layer e33ffcf7d290 3.58kB / 374B 0.8s
#23 pushing layer 736f4a42301c 64.00kB / 30.00kB 0.9s
#23 pushing layer 45cf5b5745c2 512B / 16B 2.6s
#23 pushing layer 45cf5b5745c2 3.07kB / 16B 2.6s
#23 pushing layer 233df6498cd6 3.5s done
#23 pushing layer e33ffcf7d290 3.5s done
#23 pushing layer 736f4a42301c 3.5s done
#23 pushing layer ae098c20d483 3.5s done
#23 pushing layer 468f61133dd2 512B / 933B 4.0s
#23 pushing layer cdc321148327 5.12kB / 2.61kB 4.1s
#23 pushing layer f57802063fea 22.53kB / 19.88kB 4.0s
#23 pushing layer 468f61133dd2 3.58kB / 933B 4.0s
#23 pushing layer 45cf5b5745c2 4.9s done
#23 pushing layer 7e5d8dc46f59 2.71MB / 822.41MB 5.1s
#23 pushing layer d3e42451921b 36.35kB / 2.63MB 5.3s
#23 pushing layer d3e42451921b 179.71kB / 2.63MB 5.6s
#23 pushing layer d3e42451921b 826.37kB / 2.63MB 5.8s
#23 pushing layer d3e42451921b 1.09MB / 2.63MB 6.0s
#23 pushing layer d3e42451921b 2.08MB / 2.63MB 6.2s
#23 pushing layer f57802063fea 6.2s done
#23 pushing layer 468f61133dd2 6.3s done
#23 pushing layer d3e42451921b 2.69MB / 2.63MB 6.2s
#23 pushing layer cdc321148327 6.4s done
#23 pushing layer cff0c8b3a131 23.04kB / 9.47kB 6.9s
#23 pushing layer 0a72ce9f0ac6 36.35kB / 3.05MB 6.9s
#23 pushing layer 0a72ce9f0ac6 298.50kB / 3.05MB 7.2s
#23 pushing layer 0a72ce9f0ac6 724.48kB / 3.05MB 7.3s
#23 pushing layer 330a3443e078 195.70kB / 19.03MB 7.2s
#23 pushing layer 0a72ce9f0ac6 953.86kB / 3.05MB 7.4s
#23 pushing layer 0a72ce9f0ac6 1.12MB / 3.05MB 7.5s
#23 pushing layer 0a72ce9f0ac6 1.41MB / 3.05MB 7.6s
#23 pushing layer 0a72ce9f0ac6 2.66MB / 3.05MB 7.7s
#23 pushing layer 0a72ce9f0ac6 3.05MB / 3.05MB 7.7s
#23 pushing layer 330a3443e078 1.23MB / 19.03MB 7.7s
#23 pushing layer 330a3443e078 2.20MB / 19.03MB 7.9s
#23 pushing layer 330a3443e078 4.73MB / 19.03MB 8.1s
#23 pushing layer 330a3443e078 6.31MB / 19.03MB 8.4s
#23 pushing layer 330a3443e078 7.68MB / 19.03MB 8.6s
#23 pushing layer d3e42451921b 8.8s done
#23 pushing layer 330a3443e078 9.66MB / 19.03MB 8.8s
#23 pushing layer 330a3443e078 10.65MB / 19.03MB 8.9s
#23 pushing layer 330a3443e078 11.63MB / 19.03MB 9.0s
#23 pushing layer cff0c8b3a131 9.1s done
#23 pushing layer 330a3443e078 12.62MB / 19.03MB 9.1s
#23 pushing layer 330a3443e078 14.01MB / 19.03MB 9.4s
#23 pushing layer 330a3443e078 15.40MB / 19.03MB 9.5s
#23 pushing layer 330a3443e078 16.99MB / 19.03MB 9.6s
#23 pushing layer 330a3443e078 18.21MB / 19.03MB 9.7s
#23 pushing layer 330a3443e078 19.54MB / 19.03MB 9.9s
#23 pushing layer 0a72ce9f0ac6 10.5s done
#23 pushing layer 7e5d8dc46f59 29.30MB / 822.41MB 10.6s
#23 pushing layer a3e1a9f8dd1f 10.7s
#23 pushing layer 3088d1636682 11.2s
#23 pushing layer b7d4ab99ea66 12.3s
#23 pushing layer 270a1170e7e3 12.8s
#23 pushing layer 330a3443e078 14.3s done
#23 pushing layer 7e5d8dc46f59 44.30MB / 822.41MB 15.6s
#23 pushing layer 7e5d8dc46f59 63.64MB / 822.41MB 20.7s
#23 pushing layer 7e5d8dc46f59 99.95MB / 822.41MB 26.1s
#23 pushing layer 7e5d8dc46f59 117.09MB / 822.41MB 31.5s
#23 pushing layer 7e5d8dc46f59 138.26MB / 822.41MB 36.9s
#23 pushing layer 7e5d8dc46f59 159.28MB / 822.41MB 42.3s
#23 pushing layer 7e5d8dc46f59 179.34MB / 822.41MB 47.9s
#23 pushing layer 7e5d8dc46f59 200.51MB / 822.41MB 52.9s
#23 pushing layer 7e5d8dc46f59 219.97MB / 822.41MB 58.0s
#23 pushing layer 7e5d8dc46f59 237.63MB / 822.41MB 63.1s
#23 pushing layer 7e5d8dc46f59 253.17MB / 822.41MB 68.8s
#23 pushing layer 7e5d8dc46f59 266.68MB / 822.41MB 73.9s
#23 pushing layer 7e5d8dc46f59 287.53MB / 822.41MB 79.7s
#23 pushing layer 7e5d8dc46f59 299.45MB / 822.41MB 84.8s
#23 pushing layer 7e5d8dc46f59 316.20MB / 822.41MB 90.2s
#23 pushing layer 7e5d8dc46f59 337.59MB / 822.41MB 95.2s
#23 pushing layer 7e5d8dc46f59 359.60MB / 822.41MB 100.2s
#23 pushing layer 7e5d8dc46f59 385.21MB / 822.41MB 105.3s
#23 pushing layer 7e5d8dc46f59 408.70MB / 822.41MB 110.3s
#23 pushing layer 7e5d8dc46f59 430.41MB / 822.41MB 115.4s
#23 pushing layer 7e5d8dc46f59 458.82MB / 822.41MB 120.5s
#23 pushing layer 7e5d8dc46f59 487.30MB / 822.41MB 125.6s
#23 pushing layer 7e5d8dc46f59 507.15MB / 822.41MB 130.8s
#23 pushing layer 7e5d8dc46f59 527.04MB / 822.41MB 135.8s
#23 pushing layer 7e5d8dc46f59 552.87MB / 822.41MB 140.8s
#23 pushing layer 7e5d8dc46f59 559.28MB / 822.41MB 147.7s
#23 pushing layer 7e5d8dc46f59 582.54MB / 822.41MB 152.7s
#23 pushing layer 7e5d8dc46f59 603.80MB / 822.41MB 157.8s
#23 pushing layer 7e5d8dc46f59 633.58MB / 822.41MB 162.8s
#23 pushing layer 7e5d8dc46f59 661.65MB / 822.41MB 167.8s
#23 pushing layer 7e5d8dc46f59 677.60MB / 822.41MB 173.5s
#23 pushing layer 7e5d8dc46f59 694.87MB / 822.41MB 179.0s
#23 pushing layer 7e5d8dc46f59 720.89MB / 822.41MB 184.7s
#23 pushing layer 7e5d8dc46f59 739.71MB / 822.41MB 190.6s
#23 pushing layer 7e5d8dc46f59 756.88MB / 822.41MB 196.4s
#23 pushing layer 7e5d8dc46f59 778.10MB / 822.41MB 201.5s
#23 pushing layer 7e5d8dc46f59 801.21MB / 822.41MB 206.5s
#23 pushing layer 7e5d8dc46f59 826.38MB / 822.41MB 213.0s
#23 pushing layer 7e5d8dc46f59 848.13MB / 822.41MB 218.8s
#23 pushing layer 7e5d8dc46f59 225.4s done
#23 pushing layer a3e1a9f8dd1f 230.5s done
#23 pushing layer 5f70bf18a086 230.5s done
#23 pushing layer 3088d1636682 230.5s done
#23 pushing layer b7d4ab99ea66 230.5s done
#23 pushing layer 270a1170e7e3 230.5s done
#23 DONE 231.0s
Built image, launching...
Launching server...
Server requested
2025-03-14T08:04:03.289889Z [Normal] Successfully assigned gesis/jupyter-rgaiacs-mwe4mybinder-org-y2rajd7k---356803a3 to spko-css-app03
2025-03-14T08:04:04Z [Normal] Container image "quay.io/jupyterhub/k8s-network-tools:4.0.0" already present on machine
2025-03-14T08:04:04Z [Normal] Created container: block-cloud-metadata
2025-03-14T08:04:04Z [Normal] Started container block-cloud-metadata
2025-03-14T08:04:04Z [Normal] Pulling image "gesiscss/binder-r2d-g5b5b759-rgaiacs-2dmwe4mybinder-2eorg-5c5529:08de52c3baf12b50614bd0bdd104c6fce5f7915f"