Device or Resource Busy Building Dockerfiles

[MattMencel]

Has anyone seen this or can suggest the best way to build docker images in self-hosted runners?

#25 exporting to docker image format
#25 exporting layers
#25 exporting layers 8.5s done
#25 ERROR: mount callback failed on /tmp/containerd-mount622316904: failed to unmount /tmp/containerd-mount2173409665: failed to unmount target /tmp/containerd-mount2173409665: device or resource busy
------
 > exporting to docker image format:
------
ERROR: failed to solve: mount callback failed on /tmp/containerd-mount622316904: failed to unmount /tmp/containerd-mount2173409665: failed to unmount target /tmp/containerd-mount2173409665: device or resource busy

My build steps look like this...

     - name: Set up Docker Context for Buildx
        id: buildx-context
        run: |
          docker context create builder

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          cleanup: false
          endpoint: builder

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ghcr.io/myorg/myapp/${{ matrix.name }}
          tags: |
            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
            type=ref,event=tag
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=semver,pattern={{major}}
            type=sha,format=long
            type=sha

      - name: Build and push Docker image
        uses: docker/build-push-action@v4
        with:
          build-args: |
            NPM_AUTH_TOKEN=${{ secrets.NPM_AUTH_TOKEN }}
          cache-from: type=registry,ref=ghcr.io/myorg/myapp/${{ matrix.name }}:buildcache
          cache-to: type=registry,mode=max,ref=ghcr.io/myorg/myapp/${{ matrix.name }}:buildcache
          context: .
          file: containers.d/kubernetes/${{ matrix.file }}
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

[pimguilherme]

we have also started seeing this last month, around 24th of March.. any ideas on the reason?

[sergiomacedo]

Not yet, @MattMencel. Any luck on your side?

[pimguilherme]

BTW we have been discussing this further here: moby/buildkit#3812

[sergiomacedo]

@MattMencel we dumped buildkit and changed our workflow to use docker buildx directly. So far we haven't seem any new build failure.

[MattMencel]

@sergiomacedo Are you able to share a github gist with your example?

[sergiomacedo]

@MattMencel where we had:

      - name: Setup Docker Context for buildx
        id: buildx-context
        run: |
          docker context create builders
      - name: Setup Docker buildx with local registry
        id: buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver-opts: network=host
          endpoint: builders

      - name: Build and push with local registry
        uses: docker/build-push-action@v3

We changed to:

      - name: Install buildx
        run: |
          mkdir -p ~/.docker/cli-plugins
          curl -sSL https://github.com/docker/buildx/releases/download/v0.10.3/buildx-v0.10.3.linux-amd64 -o ~/.docker/cli-plugins/docker-buildx
          chmod +x ~/.docker/cli-plugins/docker-buildx          
      - name: Build image manually
        env: 
          IMAGE_NAME: my_image_name
        run: |
          docker buildx build --cache-from type=gha --cache-to type=gha,mode=max --file Dockerfile --label mylabel=abcd123 --provenance false --tag localhost:5000/name/$IMAGE_NAME:abcd123 --builder default --pull --push .

I hope this is enough to point you in the right direction.

[adawalli]

Turning of datadog agent on the builder nodes fixes this for us - looking for a cleaner strategy now, however, take a look at your daemonsets to see what might be interfering

moby/buildkit#3812 (comment)