Trying to debug a docker in docker setup

[dannystaple]

I am using the quick start based docker compose, with the latest image.
I am using a docker-in-docker scenario, and seeing errors.
In my task config:

      - task: lint
        # image: concourse-dcind-image
        privileged: true
        config:
          platform: linux
          image_resource:
            type: docker-image
            source:
              repository: karlkfi/concourse-dcind
          inputs:
            - name: repo
          run:
            path: entrypoint.sh
            args:
              - bash
              - -ceux
              - |
                cd repo
                bin/docker-thing

Docker thing immediately performs a docker run. It's been used on the desktop. This fails in the entrypoint.sh.
made an instrumented version of entrypoint to pinpoint the failure - adding set -x. If fails where it attempts to find cgroup stuff (a bit lower level than I am with docker):

grouping="$(cat /proc/self/cgroup | cut -d: -f2 | grep "\\<${sys}\\>")"

A pipe fails, right where the grep is - since the group field has an empty string: 0::/d9435fbf-a6f5-4215-7b1a-cc7b8ba6be3c

If I add || true into the pipe, in theory, the next line should handle it being empty. However, it then fails with:

+ mount -n -t cgroup -o cpuset cgroup /sys/fs/cgroup/cpuset
mount: /sys/fs/cgroup/cpuset: permission denied.

When starting up the compose set I do see the following (which may be related):

concourse_1     | time="2022-01-12T15:44:24.101404800Z" level=error msg="failed to enable controllers ([cpuset cpu io memory hugetlb pids rdma])" error="failed to write subtree controllers [cpuset cpu io memory hugetlb pids rdma] to \"/sys/fs/cgroup/garden/cgroup.subtree_control\": write /sys/fs/cgroup/garden/cgroup.subtree_control: no such file or directory"

Concourse version: version: v7.6.0.
Running in docker desktop version Docker Desktop 4.3.2 (72729) on MacOS 11.6.2 (20G314).

Anyone got any ideas on how I resolve this?

[dannystaple]

My guess maybe that this is concourse/concourse-docker#76. I'm going to try locally override the entrypoint in my compose file with the patched one.

[dannystaple]

So I copied in the entrypoint.sh from the current concourse/concourse repo, and mounted it on top to use instead. It still failed in the same place.

The compose mount:

    volumes:
      - ./concourse_entrypoint.sh:/usr/local/bin/entrypoint.sh

In theory, I don't think I need to use the entrypoint directive as this is bind mounting on top of the containers original entrypoint.
This still has the error:

concourse_1     | time="2022-01-12T17:08:57.955329532Z" level=error msg="failed to enable controllers ([cpuset cpu io memory hugetlb pids rdma])" error="failed to write subtree controllers [cpuset cpu io memory hugetlb pids rdma] to \"/sys/fs/cgroup/garden/cgroup.subtree_control\": write /sys/fs/cgroup/garden/cgroup.subtree_control: no such file or directory"

And the job fails in the same way. Non instrumented:

selected worker: 7772f0aadb85
Setting up Docker environment...

With my instrumenting:

...
+ mountpoint -q /sys/fs/cgroup/cpuset
+ mount -n -t cgroup -o cpuset cgroup /sys/fs/cgroup/cpuset
mount: /sys/fs/cgroup/cpuset: permission denied.

[dannystaple]

I am using the yaml file mostly as suggested by https://concourse-ci.org/quick-start.html. Going to switch to the guardian-based one at https://github.com/concourse/concourse-docker/blob/master/docker-compose.yml - and see if that helps.

[dannystaple]

I would say that doesn't help:

worker_1        | {"timestamp":"2022-01-12T17:22:22.935889594Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}

[natto1784]

Did you figure something out?

[dannystaple]

Sigh - in which case it may be only a matter of downgrading to an earlier version, or waiting for concourse to release their fix (and any DinD images).

[natto1784]

Well downgrading to 7.2.0 still gives problems!

[natto1784]

Guess, will just have to wait for a fix

[dannystaple]

Ah - I mean downgrading linux to one before Cgroups v2.

[natto1784]

Ah, I've tried that. It doesn't work. It needs some kernel features to be enabled and I don't know and can't pin down which ones.

[jessielaf]

I also ran into this problem. I was not able to solve it

[Kump3r]

Try using registry-image instead of docker-image, I saw similar issues and that seemed to solve them