ERROR: mkdir /root/.docker/buildx: read-only file system

[paololazzari]

I am using binderhub and I ran into an issue with repo2docker.
When using the following configuration:

config:
  BinderHub: 
    KubernetesBuildExecutor:
      build_image: quay.io/jupyterhub/repo2docker:2024.07.0-112.g13cbc9e

when launching any repository (e.g. this one) I see the following error message in the build pod:

{"message": "Picked Git content provider.\n"}
{"message": "Cloning into '/tmp/repo2dockerha80cbgn'...\n", "phase": "fetching"}
{"message": "HEAD is now at e57d3d7 Add entrypoint.sh\n", "phase": "fetching"}
{"message": "Using DockerBuildPack builder\n", "phase": "building"}
{"message": "ERROR: mkdir /root/.docker/buildx: read-only file system\n", "phase": "building"}
{"message": "Error during build: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--file', 'Dockerfile', '--tag', 'us-central1-docker.pkg.dev/myprojectid/binderhub/binderpaololazzari-2dbinder-2dredis-2dpython-1412af:e57d3d790ec7495b39fe6c1167809dd67767f69c', '--label', 'repo2docker.version=2024.07.0+159.gd0ddd2e', '--label', 'repo2docker.repo=https://github.com/paololazzari/binder-redis-python', '--label', 'repo2docker.ref=e57d3d790ec7495b39fe6c1167809dd67767f69c', '--platform', 'linux/amd64', '/tmp/repo2dockerha80cbgn']' returned non-zero exit status 1.", "exc_info": "Traceback (most recent call last):\n  File \"/opt/venv/bin/jupyter-repo2docker\", line 8, in <module>\n    sys.exit(main())\n             ^^^^^^\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/__main__.py\", line 476, in main\n    r2d.start()\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/app.py\", line 843, in start\n    self.build()\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/app.py\", line 806, in build\n    for l in picked_buildpack.build(\n             ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/buildpacks/docker.py\", line 63, in build\n    yield from client.build(**build_kwargs)\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/docker.py\", line 160, in build\n    yield from execute_cmd(args, True)\n  File \"/opt/venv/lib/python3.12/site-packages/repo2docker/utils.py\", line 76, in execute_cmd\n    raise subprocess.CalledProcessError(ret, cmd)\nsubprocess.CalledProcessError: Command '['docker', 'buildx', 'build', '--progress', 'plain', '--push', '--build-arg', 'NB_USER=jovyan', '--build-arg', 'NB_UID=1000', '--file', 'Dockerfile', '--tag', 'us-central1-docker.pkg.dev/myprojectid/binderhub/binderpaololazzari-2dbinder-2dredis-2dpython-1412af:e57d3d790ec7495b39fe6c1167809dd67767f69c', '--label', 'repo2docker.version=2024.07.0+159.gd0ddd2e', '--label', 'repo2docker.repo=https://github.com/paololazzari/binder-redis-python', '--label', 'repo2docker.ref=e57d3d790ec7495b39fe6c1167809dd67767f69c', '--platform', 'linux/amd64', '/tmp/repo2dockerha80cbgn']' returned non-zero exit status 1.", "phase": "failed"}

I don't see this error in the previous version, 2024.07.0-89.g6d50085, but I do see it in all versions starting from 2024.07.0-112.g13cbc9e. I believe this issue was introduced by this commit jupyterhub/repo2docker@13cbc9e

[rgaiacs]

@paololazzari thanks for the report. Because you are using BinderHub, it is possible that the problem is not in repo2docker but on your Kubernetes configuration. Because of this, I would like to move your issue to the BinderHub one.

{"message": "ERROR: mkdir /root/.docker/buildx: read-only file system\n", "phase": "building"}

Are you mounting something on /root/.docker/?

[paololazzari]

this is what binderhub does when it launches a repo2docker pod:

    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-socket
    - mountPath: /root/.docker
      name: docker-config

  volumes:
  - hostPath:
      path: /var/run/docker.sock
      type: Socket
    name: docker-socket
  - name: docker-config
    secret:
      defaultMode: 420
      secretName: binder-build-docker-config

so maybe the docker-config mount breaks repo2docker?

[paololazzari]

I don't actually need to mount a secret, so I tried disabling push_secret:

config:
  BinderHub: 
    KubernetesBuildExecutor:
      build_image: quay.io/jupyterhub/repo2docker:2024.07.0-159.gd0ddd2e
    push_secret: ""

now the build works but it doesn't push the image

[manics]

Can you show us your latest full configuration and customisations? Perhaps it's related to the dynamic credentials discussed in
#1934

[paololazzari]

@manics I don't think this has anything to do with #1934. This is the configuration with which I can reproduce the error:

config.yaml:

config:
  BinderHub: 
    KubernetesBuildExecutor:
      build_image: quay.io/jupyterhub/repo2docker:2024.07.0-112.g13cbc9e
    debug: true
    use_registry: true
    base_url: /binder/
    image_prefix: us-central1-docker.pkg.dev/myprojectid/binderhub/binder
    hub_url: https://mywebsite.com/jupyterhub/
    hub_api_token: ...

secret.yaml

jupyterhub:
  hub:
    services:
      binder:
        apiToken: ...

it looks like this error is caused by this volume mount:

    - mountPath: /root/.docker
      name: docker-config

looking at the source code, this volume mount takes place under the following conditions:

binderhub/binderhub/build.py

Lines 449 to 456 in 1a445a2

	if not self.registry_credentials and self.push_secret:
	volume_mounts.append(
	client.V1VolumeMount(
	mount_path="/root/.docker/config.json",
	name="docker-config",
	sub_path="config.json",
	)
	)

in my configuration I have no registry_credentials, and push_secret is by default set to binder-build-docker-config (which is a k8s secret which for some reason was created):

binderhub/binderhub/app.py

Lines 420 to 429 in 1a445a2

	push_secret = Unicode(
	"binder-build-docker-config",
	allow_none=True,
	help="""
	DEPRECATED: Use c.BuildExecutor.push_secret
	A kubernetes secret object that provides credentials for pushing built images.
	""",
	config=True,
	)

so if I unset push_secret the build works because the volume mount does not take place. but I believe there's something wrong with the logic of the implementation here.

[yuvipanda]

@paololazzari What version of the binderhub chart are you using? I thought this should be fixed with #1929

[paololazzari]

@yuvipanda I don't see how the version of the chart is important, given that I'm overriding the build_image? Anyway, I've been using 1.0.0-0.dev.git.3673.h040c9bbe. The mount is what seems to be the problem, I think it's happening under the wrong conditions. In my case, I don't need that mount to happen altogether.