Compute job matrix distribution using extbuild matrix by smvv · Pull Request #321 · duckdb/extension-ci-tools

smvv · 2026-02-16T14:18:37Z

Context

A few problems with extension CI tools:

Ideate about how to test and iterate on this workflow.
- Idea: Generate a list of commands to run (locally or in CI) instead of hardcoding the commands in the reusable workflow YAML file.
Expensive to run most CI jobs.
- Idea: Dynamically compute matrix based on CI run environment.

As an initial step, create a small CLI tool "extbuild" to iterating quickly on these problems. For now, it can compute the job matrix. Later, we can extend the tool to generate the command list used.

Extbuild: CLI for building extensions

Usage: Cross-platform build matrix

extbuild matrix \
--platform "linux,osx,windows,wasm" \
--out $GITHUB_OUTPUT

Returns platform matrix for linux,osx,windows × amd64,arm64 and wasm × eh,mvp,threads.

The platform and arch values come from the config file config/distribution_matrix.json (by default).

The output matrix is written:

to the console for human readers.
(optionally) to --out as github output variables (thus usable for job.strategy.matrix).

Development

Install dependencies:

brew install go

Build and test extbuild:

make -C scripts/extbuild build test -sj4

The target build creates ./scripts/extbuild/build/extbuild. See ./scripts/extbuild/build/extbuild --help for local usage.

Rollout plan

The job matrix output can be slightly different from the Python script. The current rollout plan is:

Enable the tool in CI, compute the matrix, but use the old script's output.
Use the new tool's output in CI.
Remove the old script and the comparison steps.

Warn for job matrix differences

When the job matrix differs, it adds a github warning annotation:

(the step error is listed here as well, but not stopping the run due to the continue-on-error.)

and a job summary that is visible on the "run summary":

smvv · 2026-02-18T08:38:43Z

.github/workflows/_extension_distribution.yml

+            echo "=== ${platform} matrix diff (extbuild vs python) ==="
+            if ! diff -u \
+              <(jq -S 'walk(if type == "object" then del(.opt_in, .run_in_reduced_ci_mode) else . end)' "build/extbuild_ref/${platform}_matrix.json") \
+              <(jq -S 'walk(if type == "object" then del(.opt_in, .run_in_reduced_ci_mode) else . end)' "build/python_ref/${platform}_matrix.json"); then


note: This strips the json fields opt_in and run_in_reduced_ci_mode to only check the differences for the relevant JSON fields.

(The extbuild matrix command removes those fields from its JSON output lines, and thus would be missing otherwise in the diff.)

scripts/extbuild/cmd/extbuild/matrix.go

smvv · 2026-02-18T08:41:34Z

scripts/extbuild/cmd/extbuild/matrix.go

+		},
+	}
+
+	cmd.Flags().StringVar(&inputPath, "input", "config/distribution_matrix.json", "Input distribution matrix JSON file")


note: i think it's useful to rename this flag to matrix because there could be another input file perhaps in the future (not sure, but input sounds vague, for now). WDYT?

smvv · 2026-02-18T08:43:04Z

scripts/extbuild/cmd/extbuild/matrix_test.go

+func TestComputePlatformMatrices(t *testing.T) {
+	t.Parallel()
+
+	inputJSON := `{


note: this JSON string looks copied but has a reason: it makes the test stable and not dependent on the content of config/distribution_matrix.json file.

smvv · 2026-02-18T09:17:18Z

scripts/modify_distribution_matrix.py

    with open(output_json_file_path, "w") as output_json_file:
-        if filtered_data:
-            json.dump(filtered_data, output_json_file, indent=indent)
+        json.dump(filtered_data, output_json_file, indent=indent)


note: always print {} instead of empty file.

carlopi

Left a few comments, but this is solid, thanks!

A blocker for merging this it's branching (either as v1.5.0 or v1.5-variegata) a viable target for duckdb. Or anyhow deciding how to move there, but we can sort that out on a side.

carlopi · 2026-02-18T09:43:08Z

.github/workflows/_extension_distribution.yml

+            exit 1
+          fi
+
      - id: parse-matrices


Could we just do without this whole step?

Basically having a:

cp build/python_ref/* build/.

?
Idea is that we avoid duplication, stuff it's done once instead of twice, this would allow to limit weird copy paste issues (expecially if one were to modify this.

Or the other way around, where stuff it's first produced in build/, then copied to build/python_ref/ and compared, then business as usual.

Obv it's not "could", but "would it be better?"

note: the comparison and older Python script CI steps are temporary and removed later after merging this PR. Therefore, we did not bother with changing the steps further.

scripts/extbuild/cmd/extbuild/matrix.go

carlopi · 2026-02-18T09:50:32Z

scripts/extbuild/Makefile

+GO_TEST_FLAGS ?= -coverprofile=coverage.out -timeout 2s
+
+.PHONY: test cov build clean
+
+test:
+	go test ${GO_TEST_FLAGS} ./...


Should there be a top level makefile that includes this one?
Or at least, allow for that to be possible?

If so, then it would be handy to prepend like extbuild-build, extbuild-test and so.

For example, we have recently added a way to do:

make vcpkg-setup

in httpfs (https://github.com/duckdb/duckdb-httpfs?tab=readme-ov-file#building--loading-the-extension)

Idea is that httpfs includes in its own top level Makefile extension-ci-tools provided one, but for that it's handy if the commands are scoped.

Or maybe a different Makefile could be offered, that calls the commands with right path, unsure, we can chat live.

that would be useful! I think it also raises the high-level question on how we should distribute the extbuild binary to engineers. (Idea: a custom homebrew tap, precompiled binaries on github, and a github action to install it easily in CI.)

carlopi · 2026-02-19T11:06:33Z

.github/workflows/_extension_distribution.yml

+      - name: Compute extension build matrix
+        continue-on-error: true
+        run: |
+          make -C extension-ci-tools/scripts/extbuild test build -sj4


Should testing the tool also happen outside of this workflow?

Basically have a separate workflow that just does:

setup go

build & test extbuild

That could happen on a side, possibly triggered only to changes in the relevant folder.

carlopi

Thanks! I think on my side this looks nice to have, and a good starting point to build on.

I think there is no blocker in merging this from my side.

Sander Mathijs van Veen added 14 commits February 16, 2026 15:11

Try to dump event path in CI

d6e8956

Initial version of extbuild with matrix subcommand

18b4a9b

Add code coverage

a5ed317

Support wasm archs and make SplitSemicolonList private

156ad4d

Add json schema for input and output

668c2c3

Reject unknown json fields in config dist matrix

4609bcf

Display machine and human readable output

432724a

Support comma and semicolon as CLI argument separator

594da15

Support matrix subcommand without any arguments

8e3635c

Simplify tests

a185cb9

Use go 1.25 since it is installed on most CI runner images

14aa77f

Compare extbuild with script in CI (and continue on error)

033c00a

Clear any previously created build dir

3fb89ca

Make jq output human readable so matrix diffs are easy to read

94fcec1

smvv force-pushed the try-event-path branch from 8020de5 to 94fcec1 Compare February 17, 2026 15:10

Sander Mathijs van Veen added 7 commits February 17, 2026 16:29

Sort matrix include items alphabetically (easier diffing)

53f9b57

Cache fetched go modules and built binaries

2e07f37

Add --exclude flag to remove specific duckdb archs

4d6951b

Remove opt_in and run_in_reduced_ci_mode JSON fields from comparison

d10e7b3

Remove both opt_in and run_in_reduced_ci_mode from JSON comparison

7c7e100

Show matrix mismatch as warning annotation and in job summary

8a63bf8

Always output an (empty) JSON object for job matrix

552a3fa

smvv changed the title ~~Try to dump event path in CI~~ Compute job matrix distribution using extbuild matrix Feb 18, 2026

smvv requested review from carlopi and samansmink and removed request for carlopi February 18, 2026 08:35

smvv marked this pull request as ready for review February 18, 2026 08:35

smvv commented Feb 18, 2026

View reviewed changes

scripts/extbuild/cmd/extbuild/matrix.go Show resolved Hide resolved

smvv commented Feb 18, 2026

View reviewed changes

Sander Mathijs van Veen added 2 commits February 18, 2026 10:21

Move go files to scripts/extbuild to keep root dir clean

e43293c

Add README to extbuild tool

99e28ba

carlopi requested changes Feb 18, 2026

View reviewed changes

carlopi reviewed Feb 19, 2026

View reviewed changes

carlopi approved these changes Feb 19, 2026

View reviewed changes

samansmink merged commit 0de7d1d into duckdb:main Feb 19, 2026
37 checks passed

Conversation

smvv commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Extbuild: CLI for building extensions

Usage: Cross-platform build matrix

Development

Rollout plan

Warn for job matrix differences

Uh oh!

smvv Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smvv Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlopi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlopi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smvv commented Feb 16, 2026 •

edited

Loading

smvv Feb 18, 2026 •

edited

Loading

smvv Feb 18, 2026 •

edited

Loading