Docker buildx bake hangs in GitLab CI

[MalteMagnussen]

I run bake inside GitLab CI.
It builds the images, and pushes everything properly, but then just hangs.

It is inside a GitLab CI Kubernetes runner, which connects to a Remote BuildKit builder (also in k8s).

Command: docker buildx bake --builder=remote --push

Image: docker.io/library/docker:dind-rootless (no gitlab ci dind service attached)

Version inside the image at the time of usage:

/ $ docker --version
Docker version 28.3.3, build 980b856

Problem recap: Running docker buildx bake in gitlab ci with a dind rootless image in a kubernetes ci runner: The job just hangs after building and pushing, and eventually fails with a timeout.

I created a Discussion instead of a Bug Issue, because filling out the bug issue template was too overwhelming, and I cannot share all the company code and logs.

[crazy-max]

Oh sorry didn't catch this discussion, is it solved for you?

[MalteMagnussen]

@crazy-max - Sometimes it happens, sometimes it doesn't. It is currently not reproducible and very sporadic. Might be bake. Might not. Might be GitLab CI, or our buildkit server, or something else.

[MalteMagnussen]

@crazy-max - We are even seeing it if using buildctl.

So I think it has something to do with moby/buildkit + GitLab CI somehow. The "exit" signal might not be properly passed back to the CI runner. Very strange issue. Only happening here and there.

[MalteMagnussen]

It is almost exactly this to a tee we are experiencing: moby/buildkit#6131