"Supporting" buildkit/buildx

[flowchartsman]

Overview

As buildkit features like parallel builds, secrets and cache mounts become more popular, container testing frameworks like testcontainers-go should work to support them. This is something that is gaining traction for the original implementation, but has not come to pass yet.

There was an [issue](for this) with testcontainers-go once upon a time, but it evolved into the regular docker API build implementation and the original idea was dropped. This discussion seeks to resume that effort, since it would be pretty great for testcontainers-go to support buildkit, no matter how provisional, and I believe it could conceivably be staged to increase support over time.

The problems facing implementation are that buildkit does not have an HTTP api that I'm aware of. It's all over a GRPC communication with the daemon over unix socket. Compounding this, as far as I've been able to determine, Docker Desktop has their own closed GRPC API separate from the original implementation. Poking at the client on my macbooks has revealed it talking to /Users/<me>/.docker/run/docker-cli-api.sock, but it will not answer the standard moby tools like buildctl.

This doesn't have to be the end of the world, though, as most, if not all, features can be hooked via interaction with the CLI, and this should be more than enough for the initial option to simply build and tag images with buildkit instead of plain Docker, falling back to the API for the rest of the container lifecycle. Obviously, more advanced features would be better, but I think this can be done iteratively.

Phase 1

Implement an initial (and optional) BuildkitProvider type as an interface

type BuildkitProvider interface {
	CreateContainer(context.Context, ContainerRequest) (Container, error) 
}

And provide only one, implementation:

type CLIBuildkitProvider struct{}
func NewCLIBuildkitProvider() *CLIBuildkitProvider{}

This can remain opaque until such time as more options are needed and researched (such as those provided by buildx), so long as they are germane to testcontainers usage.

CLIBuildkitProvider could then be attached similarly to other options:

DockerProviderOptions struct {
        defaultBridgeNetworkName string
        buildkitProvider BuildkitProvider
        *GenericProviderOptions
}
//...
func WithBuildkitProvider(buildkitProvider BuildkitProvider) BuildkitOption {
	return BuildkitOption{
		buildkitProvider: buildkitProvider,
	}
}
//...
type BuildkitProviderOption struct {
	buildkitProvider BuildkitProvider
}

func (o BuildkitProviderOption) ApplyDockerTo(opts *DockerProviderOptions) {
	opts.buildkitProvider = o.buildkitProvider
}

If set in DockerProviderOptions, the BuildkitProvider would be delegated to to build the image. In the case of the initial CLIBuildkitProvider, this would involve calling os.Exec to run docker build with DOCKER_BUILDKIT=1 set in the environment. The CLI will also be used to label the image so that it can be claimed by the reaper.

Phase 2

Later work could include providing both

type OnDemandBuildkitProvider struct{
        //options such as retained cache location
}

as well as

type RemoteBuildkitProvider struct {
        //addr for GRPC connection
}

Taking advantage of the moby/buildkit client library to interface with the daemon over GRPC

Phase 3

• Alternate architectures via buildx, if this is somehow necessary.
• Extension to other container providers, if applicable.
• Compatibility with Docker Desktop, if such a thing is even possible.

Thoughts?

[flowchartsman]

Also, if anyone sees a need for things such as Buildah, Buildpacks, img, Kaniko, etc then the interface could be abstracted to something like

type ContainerBuilder interface {
	CreateContainer(context.Context, ContainerRequest) (Container, error) 
}

Instead.

[kiview]

I think this is a significant opportunity to create an abstraction layer. In Testcontainers for Java, we currently have a similar situation with regard to Jib integration:
https://github.com/testcontainers/testcontainers-java/pulls?q=jib

In the Testcontainers for Java architecture, this is currently planned to be achieved through the DockerClient abstraction:
https://github.com/testcontainers/testcontainers-java/pull/5623/files#diff-386dcb32939c2014f17035d347ee4c5197dcc21903d9f8cbf3008a908e47a1c3

[flowchartsman]

So, after looking into Jib, I'm further convinced that a ContainerBuilder abstraction is the right call, since this seems to be a particular niche at the moment.

Is there any reason why the Java library has opted for an impl of DockerClient as opposed to something like ContainerBuilder? From the Jib Site it definitely looks like it isn't tied to Docker, in fact, Jib explicitly mentions that it builds "without a Docker daemon".

I only ask because multi-language projects like this tend to hew (at least roughly) to the abstractions of the other so that polyglots can feel more at home with an "ecosystem". Is that the case here? Because, if not, it's a pretty strong case for ContainerBuilder

[flowchartsman]

Also: good news, it looks like Docker Desktop might respond to a client instance after all. I was able to create a container and run it with -v/Users/<me>/.docker/run/docker.sock.raw:/var/run/docker.sock and I installed the buildkit plugin and it responded to docker buildx du at least.. So that's a good sign. In theory, the client commands to issue buildkit image requests can be reverse-engineered from that.

[mdelapenya]

For some reason, I've not seen this discussion before. Sorry.

I'll go deep into it next week to understand it better.

[gheorghestrimtu]

Any updates on this work?
I am particularly interested in being able to use additional contexts, or build contexts for different stages of the build. Which seems to be supported only by BuildKit.

[adamscybot]

Alternate architectures via buildx, if this is somehow necessary.

Ok, so I am one of the folks who is touching on this unlucky use case.

I have a requirement to be able to cross-compile testcontainers targeting arm64 from an amd64 host. The reasoning is that the software under test targets arm64 in prod, and the low-level nature of that software means it's very sensitive to architecture. I want to enable realistic high-confidence testing in local development by testing against arm64, even from amd64 "developer" machines.

As of today -- the way you're led to implement this by the docker docs etc is by using buildkit-only syntax. I know there are ways to do this on pre-buildkit, but how things are in 2025 ends up making this issue more puzzling than it really is.

May I suggest a short-term change to the docs around buildOptions.Platform and buildkit generally? I spent a lot of time trying to figure out why this promising-sounding option had no effect. I eventually realised it does not affect scenarios where go-testcontainers orchestrates the image build by default since go-testcontainers simply doesn't automatically opt to use buildx. Since it calls lower level programmatic APIs to perform the build, and there's not much direction to the contrary, rabbit holes around tuning env vars like DOCKER_BUILDKIT=1 present themselves -- which relate to the first-class docker CLI only and so are ineffectual, further pedalling the incorrect notion you just can not do it. Together with this post that scores high on google, I almost begun to conclude myself you simply can't use buildx with go-testcontainers.

Subsequently, a bunch of the newer syntactical API surface of Docker around cross-compilation, multiplatform images and multi stage builds is not available by default. E.g.,by default using FROM --platform <image> syntax in the Dockerfile is ineffectual.

I don't think this is really said anywhere in a prominent way (probably because it ultimately is just surfacing some deeper programmatic docker API -- ImageBuildOptions), and it took me a while to realise. buildx, buildkit and even version on the docs page search yield no relevant results. A quick mention would sort it out, but hopefully this reply goes most of the way.

🎉 Yes you can already use buildkit with go-testcontainers and fix these issues today! 🎉:

	from := tc.FromDockerfile{
	       // ...
		BuildOptionsModifier: func(buildOptions *build.ImageBuildOptions) {
			buildOptions.Platform = "linux/arm64"
			buildOptions.Version = build.BuilderBuildKit // <-- You need this!
		},
	}

Now I can do everything I need, and the change to buildOptions.Platform in my example is effective. Presumably this also enables the newer syntax/approach for multi stage builds etc.

I also think this is a big enough deal that perhaps there should be a first-class BuildKit bool property directly addressable on the FromDockerfile struct, without using BuildOptionsModifier. Thus, helping to solve the discoverability issue. This is a similar approach to how other loosely related libs have solved it. It being obscure by BuildOptionsModifier at the same time as being ambiguously named Version means I'm not the only one who had no idea.

Many have also just made it the default (e.g goreleaser's docker capabilities), and perhaps that's the ultimate fix and the most appropriate one given buildkit maturity. Albeit, one that probably needs a major version bump to put in place.

Sorry for the annoying ping, but I'm going to tag mdelapenya for thoughts on above suggestions as its potentially high-impact low-hanging-fruit judging by upvotes across this thread/comments, and I was minutes away from giving up as well before discovering it.

[mdelapenya]

@adamscybot thanks for such a detailed response about the state of the issue. I got to the same point here #2482 (comment), having a dedicated module that enabled buildkit using the version. @stevenh found some issues, althoug we never went back to it since then, so the project has changed.

Nevertheless, I can share that having a programmatic API for buildkit is in the internal conversations here at Docker, so I'll link your post with the different teams, to help in the prioritisation.

[mabrarov]

Hi,

This solution seems to be insufficient, because, if base image is missing, build of image (FromDockerfile) fails (reproduced on Linux, didn't reproduce on macOS) during pulling of base image with no active sessions error. Refer to moby/moby#48112.

Thank you.