Investigate the integration of Hermeto and ko-builder

[brunoapimentel]

In order to make the ko-builder have feature-parity with the buildah tasks, one of the things needed is make it able to handle hermetic builds.

In other words, the ko-builder task needs to be able to prefetch the base image, use Hermeto to prefetch the Go necessary modules, and execut ko build in complete network isolation.

[brunoapimentel]

In my preliminary investigation, I wasn't able to make ko use the env variables produced by Hermeto. Maybe I got something wrong in my .ko.yaml file.

Below is a simple reproducer. Notice that the fetching of the base image and setting the KOCACHE env var works as expected.

#!/bin/bash

TARGET_REPO=https://github.com/konflux-ci/build-service
HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest
KO_IMAGE=quay.io/redhat-user-workloads/konflux-build-pipeli-tenant/ko-builder:ko-builder-on-pull-request-l95x6-build-container
BASE_IMAGE=registry.access.redhat.com/ubi9/go-toolset@sha256:6a73e8f2d2433b4b71127970be5afc14c19144a03daef0031364734a6b03672a

TMP_DIR=$(mktemp -d "/tmp/kobuilder.XXXXXXXXXX")

set -ex

# clone target repository
cd $TMP_DIR
git clone "$TARGET_REPO" sources

mkdir output

# use Hermeto to prefetch dependencies
podman run --rm \
	-v $(realpath ./sources):/tmp/sources:z \
	-v $(realpath ./output):/tmp/output:z \
	"$HERMETO_IMAGE" \
	--log-level "DEBUG" \
	fetch-deps gomod \
	--source "/tmp/sources" \
	--output "/tmp/output"

docker run -it --rm -v $(pwd):/workspace help

# use oras to prefetch the base image
mkdir -p cache/img

podman run --rm \
    -v $(realpath ./cache/img):/workspace:z
    ghcr.io/oras-project/oras:v1.3.0 \
    copy "$BASE_IMAGE" --to-oci-layout /workspace

# create a .ko.yaml config file
# the env variables here should be created by Hermeto, but we're hardcoding them here for simplicity
cat << EOF > ./sources/.ko.yaml
defaultBaseImage: $BASE_IMAGE
builds:
- id: sample
  dir: .
  main: ./cmd/main.go
  env:
  - GOCACHE=/tmp/output/deps/gomod
  - GOMODCACHE=/tmp/output/deps/gomod/pkg/mod
  - GOPATH=/tmp/output/deps/gomod
  - GOPROXY=file:///tmp/output/deps/gomod/pkg/mod/cache/download
EOF

# run the ko build
podman run --rm \
    --user 0 \
    -v $(realpath ./sources):/tmp/sources:z \
    -v $(realpath ./output):/tmp/output:z \
    -v $(realpath ./cache):/tmp/cache:z \
    -e KO_DOCKER_REPO=ko.local \
    -e KOCACHE=/tmp/cache \
    --network=none \
    --workdir=/tmp/sources \
    "$KO_IMAGE" \
    build -v --push=false cmd/main.go

[vdemeester]

@brunoapimentel so I used crane for it, and I managed to make something work:

$ crane pull --format oci ghcr.io/tektoncd/plumbing/tini-git@sha256:2c18f0b3ed4394e27068b5c70bb55419797e8fc743d8ea9e0c2766001b36b5b4 $KOCACHE/img
$ tree $KOCACHE/img       
/home/vincent/.local/cache/ko/img
├── blobs
│   └── sha256
│       ├── 2c18f0b3ed4394e27068b5c70bb55419797e8fc743d8ea9e0c2766001b36b5b4
│       ├── 32f85ef326f5d9aed7a3379aacaa688df61904b14156146c9f2f647e1471b494
│       ├── 33a2433d89df7e794d1655fce70d7031d8065c9798bdc2931f7c98fcc8d310d0
│       ├── 3b923968bfaeec6b1fffa40acc1ad1aa3e5584321396a5b2b8f93b61865ad44b
│       ├── 3e18f3ba6afc5c925849b5dad50986fdfcadc029086a1c7231cd9b0749abfaef
│       ├── 446aed612d8c1a492e3d0e451177dd0426746c8f907d98dd0a25896d3b2f6ddc
│       ├── 47a70fdc8ac4c1273de626dc7710d3e19cfd5b9f3e10cfc4b14602bdfffbffe1
│       ├── 54668509cac0acfce868cd4c7ae6103ccc3cec445f4c999b35ae450412967c5e
│       ├── 5a8d3b6ca53795a9e87ceec6068eef348121f20406b5db105c5da6d580f7ebec
│       ├── 5fc47bfc2b4c51183b15ada069ea9e14e5fa1eeeeeb7a20380a6d158e5c34e63
│       ├── 633efdbb109d2dc76326d2280528b9d357915ac6043a19b4bdea6ec170805715
│       ├── 656243eafd6d070ee48908d86facf0c76c2d84d87e5656fad77d1da697d6d5a3
│       ├── 68f5a00f978162fa15f366da9fecf585750c1f8f143dda2adcc90f31272c3058
│       ├── 6be119eaea2985d82802ffd519f0955329bcead0258ed6678e87b992b3de7154
│       ├── 6d8c24d6f37a696a32b52f5cfc85999f7b98351f9f351b77853575432e436d65
│       ├── 8aac200ff046e81eacad9097167ff55c14aea9de2303c40534fc43d66c7ac89b
│       ├── 90f3c286297905b48c9b6672a719a2c16f3d1b6ef8ea182485c643c834c059b4
│       ├── 939e9db77e6425a5c0efe7a34aa0da2e613968d0566a262720870c1c3dfdd7de
│       ├── 96560ac855d5dfdccea74c592e3dd5feb2c4867c655b78b936847992c95a6d86
│       ├── a3ca93737924e56b0504fc25853be24795cdd95de3aea30489d5556712d9c02a
│       ├── b09c86b9fe43bd4cb71eca19d69f656c874e7da865f9e87994654cca5a787acb
│       ├── c33858286711c49fe7b9c82f0881a7aa6576ed23a45b3bfca751cffe5179cbed
│       ├── c6f1be9f5980ea999fc701fb87352d837133ccf6f35e125969552f37acff4d05
│       ├── cf74cbf0c74e720427c500e6a597f21570361268f36fa74c520ac1aa2c9a9983
│       ├── cfc14afccee6e41c64c623abead729adb3074f0d453b36876f9fc60f17fc0f36
│       ├── d69d4d41cfe2ee680d6972795e2a1eb9e4dc4ec3b3c5e0797c9ab43bb3726fa7
│       ├── f4d2bd3faf3efd232c936620886a1868a8f974fb01f58e2c8ca4a65ccab2bf8c
│       ├── fca31cd68add90bca30a3bf0501856883848e0c79f3c1d791a7872df44368441
│       └── fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236
├── index.json
└── oci-layout

3 directories, 31 files
$ unshare -n -r ko build --push=false ./cmd/resolvers
2025/10/14 15:45:31 Using base ghcr.io/tektoncd/plumbing/tini-git@sha256:2c18f0b3ed4394e27068b5c70bb55419797e8fc743d8ea9e0c2766001b36b5b4 for github.com/tektoncd/pipeline/cmd/resolvers
[...]
2025/10/14 15:45:31 Building github.com/tektoncd/pipeline/cmd/resolvers for linux/amd64
ttl.sh/resolvers-ff86b24f130c42b88983d3c13993056d:latest@sha256:e82ea2bd3f5a28510e38cd72c3c5ac6b47421aa3c81077b92aa3dab0d1ed0d58

In your example, where does it fail ? (running it as I am writing it)

[vdemeester]

I am getting go proxy error which means somehow the go modules are not "cached". But I am not sure about your .ko.yaml though, it doesn't look correct to me.

[vdemeester]

#!/bin/bash

TARGET_REPO=https://github.com/konflux-ci/build-service
HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest
KO_IMAGE=quay.io/redhat-user-workloads/konflux-build-pipeli-tenant/ko-builder:ko-builder-on-pull-request-l95x6-build-container
BASE_IMAGE=registry.access.redhat.com/ubi9/go-toolset@sha256:6a73e8f2d2433b4b71127970be5afc14c19144a03daef0031364734a6b03672a

TMP_DIR=$(mktemp -d "/tmp/kobuilder.XXXXXXXXXX")

set -ex

# clone target repository
cd $TMP_DIR
git clone "$TARGET_REPO" sources

mkdir output

# use Hermeto to prefetch dependencies
podman run --rm \
	-v $(realpath ./sources):/tmp/sources:z \
	-v $(realpath ./output):/tmp/output:z \
	"$HERMETO_IMAGE" \
	--log-level "DEBUG" \
	fetch-deps gomod \
	--source "/tmp/sources" \
	--output "/tmp/output"

# use oras to prefetch the base image
mkdir -p cache/img

podman run --rm \
    -v $(realpath ./cache/img):/workspace:z \
    ghcr.io/oras-project/oras:v1.3.0 \
    copy "$BASE_IMAGE" --to-oci-layout /workspace

# create a .ko.yaml config file
# the env variables here should be created by Hermeto, but we're hardcoding them here for simplicity
cat << EOF > ./sources/.ko.yaml
defaultBaseImage: $BASE_IMAGE
EOF

# run the ko build
podman run --rm \
    --user 0 \
    -v $(realpath ./sources):/tmp/sources:z \
    -v $(realpath ./output):/tmp/output:z \
    -v $(realpath ./cache):/tmp/cache:z \
    -e KO_DOCKER_REPO=ttl.sh \
    -e KOCACHE=/tmp/cache \
    -e GOCACHE="/tmp/output/deps/gomod/pkg/mod"\
    -e GOMODCACHE="/tmp/output/deps/gomod/pkg/mod" \
    -e GOPATH="/tmp/output/deps/gomod" \
    -e GOPROXY="file:///tmp/output/deps/gomod/pkg/mod/cache/download" \
    --network=none \
    --workdir=/tmp/sources \
    "$KO_IMAGE" \
    build -v --push=false cmd/main.go

@brunoapimentel this almost works for me, until pushing to docker (but running in podman, it is expected). I think there is a weird case with ko.local and the push flag. If you use any other KO_REPO (like ttl.sh) it just work.

[vdemeester]

And we can use --oci-layout-path or --tarball and --sbom-dir if we want to push things somewhere after the build.

     --oci-layout-path string     Path to save the OCI image layout of the built images
      --sbom-dir string            Path to file where the SBOM will be written.
      --tarball string             File to save images tarballs

[brunoapimentel]

#!/bin/bash

TARGET_REPO=https://github.com/konflux-ci/build-service
HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest
KO_IMAGE=quay.io/redhat-user-workloads/konflux-build-pipeli-tenant/ko-builder:ko-builder-on-pull-request-l95x6-build-container
BASE_IMAGE=registry.access.redhat.com/ubi9/go-toolset@sha256:6a73e8f2d2433b4b71127970be5afc14c19144a03daef0031364734a6b03672a

TMP_DIR=$(mktemp -d "/tmp/kobuilder.XXXXXXXXXX")

set -ex

# clone target repository
cd $TMP_DIR
git clone "$TARGET_REPO" sources

mkdir output

# use Hermeto to prefetch dependencies
podman run --rm \
	-v $(realpath ./sources):/tmp/sources:z \
	-v $(realpath ./output):/tmp/output:z \
	"$HERMETO_IMAGE" \
	--log-level "DEBUG" \
	fetch-deps gomod \
	--source "/tmp/sources" \
	--output "/tmp/output"

# use oras to prefetch the base image
mkdir -p cache/img

podman run --rm \
    -v $(realpath ./cache/img):/workspace:z \
    ghcr.io/oras-project/oras:v1.3.0 \
    copy "$BASE_IMAGE" --to-oci-layout /workspace

# create a .ko.yaml config file
# the env variables here should be created by Hermeto, but we're hardcoding them here for simplicity
cat << EOF > ./sources/.ko.yaml
defaultBaseImage: $BASE_IMAGE
EOF

# run the ko build
podman run --rm \
    --user 0 \
    -v $(realpath ./sources):/tmp/sources:z \
    -v $(realpath ./output):/tmp/output:z \
    -v $(realpath ./cache):/tmp/cache:z \
    -e KO_DOCKER_REPO=ttl.sh \
    -e KOCACHE=/tmp/cache \
    -e GOCACHE="/tmp/output/deps/gomod/pkg/mod"\
    -e GOMODCACHE="/tmp/output/deps/gomod/pkg/mod" \
    -e GOPATH="/tmp/output/deps/gomod" \
    -e GOPROXY="file:///tmp/output/deps/gomod/pkg/mod/cache/download" \
    --network=none \
    --workdir=/tmp/sources \
    "$KO_IMAGE" \
    build -v --push=false cmd/main.go

@brunoapimentel this almost works for me, until pushing to docker (but running in podman, it is expected). I think there is a weird case with ko.local and the push flag. If you use any other KO_REPO (like ttl.sh) it just work.

I didn't have time to properly test this, but the first thing I noticed is that you dropped the Hermeto env variables, so I wonder how the build succeeded in an airgapped container (notice there's no vendoring in the target repo) 🤔.

Any chance some build cache is being picked up? It doesn't seem like it, but still. I will try to get some time to investigate this further.

[vdemeester]

I didn't have time to properly test this, but the first thing I noticed is that you dropped the Hermeto env variables, so I wonder how the build succeeded in an airgapped container (notice there's no vendoring in the target repo) 🤔.

@brunoapimentel they are not dropped, they are passed as environment variables in the podman command. ko.yaml is a very bare and little configuration file. Essentially the task should export those env (or source a file) when running and it will just work.

    -e KOCACHE=/tmp/cache \
    -e GOCACHE="/tmp/output/deps/gomod/pkg/mod"\
    -e GOMODCACHE="/tmp/output/deps/gomod/pkg/mod" \
    -e GOPATH="/tmp/output/deps/gomod" \
    -e GOPROXY="file:///tmp/output/deps/gomod/pkg/mod/cache/download" \

Given how it is done for buildah, my guess is that either the step or the command ko should be run hermetically (using unshare). But if we set the env variable to the right place (GOCACHE, GOMODCACHE, ..) it will work.

[brunoapimentel]

brunoapimentel they are not dropped, they are passed as environment variables in the podman command. ko.yaml is a very bare and little configuration file. Essentially the task should export those env (or source a file) when running and it will just work.

Oh, I'm sorry for having overlooked that. It's good to know that ko simply picks up the environment variables without needing to cram them into .ko.yaml. We can probably drop the .ko.yaml creation and replace it with KO_DEFAULTBASEIMAGE then.

[brunoapimentel]

In Konflux, we might be able to export the image to a volume using the --oci-layout-path option. Here's my newest working version of the script:

#!/bin/bash

TARGET_REPO=https://github.com/konflux-ci/build-service
HERMETO_IMAGE=quay.io/konflux-ci/hermeto:latest
KO_IMAGE=quay.io/redhat-user-workloads/konflux-build-pipeli-tenant/ko-builder:ko-builder-on-pull-request-l95x6-build-container
BASE_IMAGE=registry.access.redhat.com/ubi9/go-toolset@sha256:6a73e8f2d2433b4b71127970be5afc14c19144a03daef0031364734a6b03672a

TMP_DIR=$(mktemp -d "/tmp/kobuilder.XXXXXXXXXX")

set -ex

# clone target repository
cd $TMP_DIR
git clone "$TARGET_REPO" sources

mkdir output

# use Hermeto to prefetch dependencies
podman run --rm \
	-v $(realpath ./sources):/tmp/sources:z \
	-v $(realpath ./output):/tmp/output:z \
	"$HERMETO_IMAGE" \
	--log-level "DEBUG" \
	fetch-deps gomod \
	--source "/tmp/sources" \
	--output "/tmp/output"

# use oras to prefetch the base image
mkdir -p cache/img

podman run --rm \
    -v $(realpath ./cache/img):/workspace:z \
    ghcr.io/oras-project/oras:v1.3.0 \
    copy "$BASE_IMAGE" --to-oci-layout /workspace

# create a directory to store the built image
mkdir containers

# run the ko build
podman run --rm \
    --user 0 \
    -v $(realpath ./sources):/tmp/sources:z \
    -v $(realpath ./output):/tmp/output:z \
    -v $(realpath ./cache):/tmp/cache:z \
    -v $(realpath ./containers):/tmp/containers:z \
    -e KO_DEFAULTBASEIMAGE="$BASE_IMAGE" \
    -e KO_DOCKER_REPO=ttl.sh \
    -e KOCACHE=/tmp/cache \
    -e GOCACHE="/tmp/output/deps/gomod/pkg/mod"\
    -e GOMODCACHE="/tmp/output/deps/goKO_DEFAULTBASEIMAGEmod/pkg/mod" \
    -e GOPATH="/tmp/output/deps/gomod" \
    -e GOPROXY="file:///tmp/output/deps/gomod/pkg/mod/cache/download" \
    --network=none \
    --workdir=/tmp/sources \
    "$KO_IMAGE" \
    build -v --push=false --oci-layout-path=/tmp/containers cmd/main.go