GESIS server reports "no space left on device"

[rgaiacs]

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2docker_sxxe4sq'...
HEAD is now at e409473 fixed typo in install.R
Python version unspecified, using current default Python version 3.10. This will change in the future.Building conda environment for python=3.10
Using RBuildPack builder
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 ERROR: failed to prepare  as x2213k2asjynz7yk9gmb8npix: mkdir /var/lib/docker/overlay2/x2213k2asjynz7yk9gmb8npix: no space left on device
------
 > [internal] load build definition from Dockerfile:
------
ERROR: failed to solve: ResourceExhausted: ResourceExhausted: ResourceExhausted: failed to read dockerfile: failed to prepare  as x2213k2asjynz7yk9gmb8npix: mkdir /var/lib/docker/overlay2/x2213k2asjynz7yk9gmb8npix: no space left on device
Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--tag', 'gesiscss/binder-r2d-g5b5b759-gesiscss-2djupyter4nfdi-5fsurvey-5fresults-727aee:e4094732f9406561080efb7b652fe19711bc03b1', '--platform', 'linux/amd64', '/tmp/tmpkfim6nlm']' returned non-zero exit status 1.

[manics]

Could it be related to
#3249
?

[rgaiacs]

This is caused due to some malfunction of image-cleaner pod. The parameters for the image-cleaner are

imageGCThresholdHigh: 1600e9
imageGCThresholdLow: 1000e9
imageGCThresholdType: absolute

but the image-cleaner pod is not removing the old images

2025-03-10 06:51:35,415 2201.64GB used
2025-03-10 06:51:35,425 No images to delete
2025-03-10 07:22:49,380 2201.64GB used
2025-03-10 07:22:49,389 No images to delete
2025-03-10 07:54:08,132 2201.64GB used
2025-03-10 07:54:08,141 No images to delete

[rgaiacs]

Could it be related to #3249

Yes, it is probably related to it. But the GESIS chart has

enabled: true

[rgaiacs]

This was temporarily resolved by cleaning the mount volume used to store the builds. This needs a more permanent solution.

[yuvipanda]

I think this is a manifestation of #3249 (comment). I would say jupyterhub/binderhub#1940 is the 'permanent' solution here.

In the meantime, perhaps we can swap out the cronjob buildkit cleaner to be a sidecar of the dind pod instead?

[rgaiacs]

In the meantime, perhaps we can swap out the cronjob buildkit cleaner to be a sidecar of the dind pod instead?

My understanding of the existing optimisation is that for the minimal working example

FROM alpine:3.21.3

RUN apk add py3-numpy

COPY . .

we want to keep the 1st (alpine) and 2nd layer (py3-numpy) in disk and clean the 3rd layer (copy).

My inexperience with sidecar makes difficult for me to understand how sidecar will help us and preserve the existing optimisation.

[rgaiacs]

This is happening again. I will manually delete the data one more time as we will soon replace the server as mentioned in #3264.

[rgaiacs]

This is not relevant after #3267.