kubernetes

Kubernetes manifests for BuildKit

This directory contains Kubernetes manifests for Pod, Deployment (with Service), StatefulSet, and Job.

• Pod: good for quick-start
• Deployment + Service: good for random load balancing with registry-side cache
• StateFulset: good for client-side load balancing, without registry-side cache
• Job: good if you don't want to have daemon pods

Variants

• *.privileged.yaml: Launches the Pod as the fully privileged root user.
• *.rootless.yaml: Launches the Pod as a non-root user, whose UID is 1000.
• *.userns.yaml: Launches the Pod as a non-root user. The UID is determined by kubelet. Needs kubelet and kube-apiserver to be reconfigured to enable the UserNamespacesSupport feature gate.

It is recommended to use *.rootless.yaml to minimize the chance of container breakout attacks.

See also:

• ../../docs/rootless.md.
• "Building Images Efficiently And Securely On Kubernetes With BuildKit" (KubeCon EU 2019).

Pod

kubectl apply -f pod.rootless.yaml

buildctl \
  --addr kube-pod://buildkitd \
  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir

If rootless mode doesn't work, try pod.privileged.yaml.

⚠️ kube-pod:// connection helper requires Kubernetes role that can access pods/exec resources. If pods/exec is not accessible, use Service instead (See below).

Deployment + Service

Setting up mTLS is highly recommended.

./create-certs.sh SAN [SAN...] can be used for creating certificates.

./create-certs.sh 127.0.0.1

The daemon certificates is created as Secret manifest named buildkit-daemon-certs.

kubectl apply -f .certs/buildkit-daemon-certs.yaml

Apply the Deployment and Service manifest:

kubectl apply -f deployment+service.rootless.yaml

kubectl scale --replicas=10 deployment/buildkitd

Run buildctl with TLS client certificates:

kubectl port-forward service/buildkitd 1234

buildctl \
  --addr tcp://127.0.0.1:1234 \
  --tlscacert .certs/client/ca.pem \
  --tlscert .certs/client/cert.pem \
  --tlskey .certs/client/key.pem \
  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir

StatefulSet

StatefulSet is useful for consistent hash mode.

kubectl apply -f statefulset.rootless.yaml
kubectl scale --replicas=10 statefulset/buildkitd
buildctl \
  --addr kube-pod://buildkitd-4 \
  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir

See ./consistenthash for how to use consistent hashing.

Job

kubectl apply -f job.rootless.yaml

To push the image to the registry, you also need to mount ~/.docker/config.json and set $DOCKER_CONFIG to /path/to/.docker directory.