feature: add duckdb support #365

James-Gilbert- · 2025-06-10T23:23:15Z

Changes

Adds basic support for DuckDB in file based mode using go-duckdb with the x_bento_extra tag
Upgrades to Go 1.24

Testing

I verified some basic functionality with reading/exporting to parquet by mounting a writeable volume. According to the go-duckdb docs, JSON + Parquet support should be included in the binaries.
Purely in-memory behavior wasn't tested. As far as I can tell, there is some remaining effort required for safe concurrent access via the go bindings.

Notes

Due to some ABI compatibility issues with the pre-built static binaries, a newer base image and changes to the Dockerfile.cgo were needed. It is possible to build each of the dependencies from source (either dynamically or statically).

From experiments, it's possible to get a statically linked binary down to ~200MB building from source, with build time and licensing requirements being the primary trade-off.

gregfurman

Wow thanks for the PR! Have left some comments.

I was giving the original thread and linked issues a read. It appears that duckdb v1.3.1 will be fixing this. See marcboeker/go-duckdb#454 (comment)

Should we perhaps just wait until this has been fixed and see if that'll solve the CGO dependency? Would be great to have this in the pure Go bento build 😄

gregfurman · 2025-06-14T12:14:58Z

internal/impl/sql/conn_fields.go

 }

 func sqlOpenWithReworks(ctx context.Context, logger *service.Logger, driver, dsn string, connSettings *connSettings) (*sql.DB, error) {
+	if !slices.Contains(sql.Drivers(), driver) {


I really like this approach! I think, however, that we should leverage build tags to dynamically set how we should be validating this. For example, see the driverField variable above at

bento/internal/impl/sql/conn_fields.go

Lines 17 to 18 in 9fc0fcd

var driverField = service.NewStringEnumField("driver", "mysql", "postgres", "clickhouse", "mssql", "sqlite", "oracle", "snowflake", "trino", "gocosmos", "spanner", "duckdb").

Description("A database [driver](#drivers) to use.")

What if instead we added a LintRule that (by default) returned this error message you've added below except, we only set this when the x_bento_extra tag has NOT been used.

So we add a conn_fields_pure.go where we overwrite this driverField to raise a lint error when not in CGO/bento_extra:

//go:build !x_bento_extra package sql func init() { driverField = service.NewStringEnumField("driver", "mysql", "postgres", "clickhouse", "mssql", "sqlite", "oracle", "snowflake", "trino", "gocosmos", "spanner"). Description("A database [driver](#drivers) to use.").LintRule(`root = if this == "duckdb" { [ "Cannot use DuckDB driver outside of CGO build. See ..." ] }`) }

internal/impl/sql/integration_extras_test.go

James-Gilbert- · 2025-06-29T20:06:18Z

Wow thanks for the PR! Have left some comments.

I was giving the original thread and linked issues a read. It appears that duckdb v1.3.1 will be fixing this. See marcboeker/go-duckdb#454 (comment)

Should we perhaps just wait until this has been fixed and see if that'll solve the CGO dependency? Would be great to have this in the pure Go bento build 😄

Due to C++ deps, CGO builds are needed for duckdb. But there are a few strategies to reduce overall image and/or binary size.

gregfurman · 2025-07-02T15:53:33Z

Can you run a make docs and commit the updated.md files?

gregfurman · 2025-07-02T16:13:39Z

internal/impl/sql/conn_fields_init.go

+func init() {
+	driverField = service.NewStringEnumField("driver", "mysql", "postgres", "clickhouse", "mssql", "sqlite", "oracle", "snowflake", "trino", "gocosmos", "spanner").
+		Description("A database [driver](#drivers) to use.").LintRule(`root = if this == "duckdb" { [ "Cannot use DuckDB driver outside of CGO build. Use a binary built with tag x_bento_extra, or the CGO Docker image" ] }`)
+}


Tested this locally and all looks good!

gregfurman · 2025-07-02T16:23:57Z

resources/docker/Dockerfile.cgo

@@ -1,36 +1,72 @@
-FROM golang:1.23 AS build
+FROM ubuntu:24.04 AS build


Can we not stay on an official golang image when building?

Might be able to now with the duckdb update, I had a PR there that downgraded the C++ ABI for better compatibility.

OK sick. Have these upstream changes been merged? Or should we address in a follow up

gregfurman · 2025-07-02T16:24:28Z

resources/docker/Dockerfile.cgo


 # Pack
-FROM debian:latest
+FROM ubuntu:24.04 AS final


Similar question as above: why the move over to ubuntu?

Needed a newer C++ runtime environment for the duckdb dependencies since it's not a fully static build

gregfurman · 2025-07-02T16:28:11Z

resources/docker/Dockerfile.cgo

+# Copy the full source and build
+COPY . ./


Wont this also copy the entire website/, .git/, resources/ etc.?

Yeah, has the same behavior as before but uses the relative path.

gregfurman · 2025-07-02T16:28:41Z

resources/docker/Dockerfile.cgo

+
+# Install minimal runtime dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libzmq5 ca-certificates \


Why install ca-certificates both in builder and again in final?

This was a holdover from trying to use slimmer images, can just copy it over again

James-Gilbert-

To clarify, is the goal of the build process to produce a fully static binary? Or is it to create an image with the extra CGO runtime dependencies?

James-Gilbert- · 2025-07-09T18:26:13Z

resources/docker/Dockerfile.cgo

+# Copy the full source and build
+COPY . ./


Yeah, has the same behavior as before but uses the relative path.

James-Gilbert- · 2025-07-09T18:27:36Z

resources/docker/Dockerfile.cgo


 # Pack
-FROM debian:latest
+FROM ubuntu:24.04 AS final


Needed a newer C++ runtime environment for the duckdb dependencies since it's not a fully static build

James-Gilbert- · 2025-07-09T18:29:53Z

resources/docker/Dockerfile.cgo

+
+# Install minimal runtime dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libzmq5 ca-certificates \


This was a holdover from trying to use slimmer images, can just copy it over again

James-Gilbert- · 2025-07-10T18:16:52Z

resources/docker/Dockerfile.cgo

@@ -1,36 +1,72 @@
-FROM golang:1.23 AS build
+FROM ubuntu:24.04 AS build


Might be able to now with the duckdb update, I had a PR there that downgraded the C++ ABI for better compatibility.

gregfurman · 2025-07-14T16:11:20Z

Or is it to create an image with the extra CGO runtime dependencies?

@James-Gilbert- I suppose that since we don't release the CGO/x_bento_extra builds as a static binary, the goal would be to create an image with runtime dependencies baked in.

To clarify, is the goal of the build process to produce a fully static binary?

Curious, would something like this be possible? Perhaps using embed with those C dependencies being statically linked. I'm no expert here so excuse the ignorance 🤷

gregfurman · 2025-07-14T16:12:37Z

Also, lmk if you need me to re-build the new SQL documentation since it's failing in your CI 😄

James-Gilbert- · 2025-07-21T19:40:37Z

Or is it to create an image with the extra CGO runtime dependencies?

@James-Gilbert- I suppose that since we don't release the CGO/x_bento_extra builds as a static binary, the goal would be to create an image with runtime dependencies baked in.

To clarify, is the goal of the build process to produce a fully static binary?

Curious, would something like this be possible? Perhaps using embed with those C dependencies being statically linked. I'm no expert here so excuse the ignorance 🤷

I think it'd be possible to statically-cross compile, but with significant effort (probably some upstream effort w/DuckDB). Having a final layer with runtime dependencies is much less of a change.

gregfurman · 2025-07-21T20:24:08Z

Or is it to create an image with the extra CGO runtime dependencies?

@James-Gilbert- I suppose that since we don't release the CGO/x_bento_extra builds as a static binary, the goal would be to create an image with runtime dependencies baked in.

To clarify, is the goal of the build process to produce a fully static binary?

Curious, would something like this be possible? Perhaps using embed with those C dependencies being statically linked. I'm no expert here so excuse the ignorance 🤷

I think it'd be possible to statically-cross compile, but with significant effort (probably some upstream effort w/DuckDB). Having a final layer with runtime dependencies is much less of a change.

Gotcha! Once you run a make docs and commit the generated files, it should unblock the CI 🟢

Otherwise, lmk re: your capacity to address the remainder of image comments. I'm happy to also follow up if you'd prefer 🙂

James-Gilbert- · 2025-07-23T16:51:04Z

It seems that because of the build tag init(), the "duckdb" driver string is dropped after re-running make docs

internal/impl/sql/conn_fields_init.go

aronchick · 2025-12-24T19:12:37Z

would really love to have this! lmk if i can help at all

adding duckdb support

a9ea781

James-Gilbert- requested review from gregfurman and jem-davies as code owners June 10, 2025 23:23

James-Gilbert- mentioned this pull request Jun 10, 2025

Adding support for DuckDB #326

Open

gregfurman assigned James-Gilbert- Jun 12, 2025

docs + linting fix

9fc0fcd

gregfurman reviewed Jun 14, 2025

View reviewed changes

James-Gilbert- and others added 2 commits June 29, 2025 15:57

incorporated feedback from review

e28be69

Merge branch 'main' into feat/duckdb-support

aa89380

negated build tag

3b06ba6

gregfurman reviewed Jul 2, 2025

View reviewed changes

James-Gilbert- commented Jul 10, 2025

View reviewed changes

rebuilt docs

7a9aea8

gregfurman reviewed Jul 30, 2025

View reviewed changes

internal/impl/sql/conn_fields_init.go Show resolved Hide resolved

gregfurman added this to the 1.11 milestone Aug 28, 2025

chore: update docs

867c358

gregfurman modified the milestones: 1.11, 1.13 Oct 22, 2025

sapk mentioned this pull request Jan 26, 2026

feature: add duckdb support #674

Open

	var driverField = service.NewStringEnumField("driver", "mysql", "postgres", "clickhouse", "mssql", "sqlite", "oracle", "snowflake", "trino", "gocosmos", "spanner", "duckdb").
	Description("A database [driver](#drivers) to use.")

		@@ -1,36 +1,72 @@
		FROM golang:1.23 AS build
		FROM ubuntu:24.04 AS build

feature: add duckdb support #365

Are you sure you want to change the base?

feature: add duckdb support #365

Uh oh!

Conversation

James-Gilbert- commented Jun 10, 2025

Changes

Testing

Notes

Uh oh!

gregfurman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

James-Gilbert- commented Jun 29, 2025

Uh oh!

gregfurman commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

James-Gilbert- left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregfurman commented Jul 14, 2025

Uh oh!

gregfurman commented Jul 14, 2025

Uh oh!

James-Gilbert- commented Jul 21, 2025

Uh oh!

gregfurman commented Jul 21, 2025

Uh oh!

James-Gilbert- commented Jul 23, 2025

Uh oh!

Uh oh!

aronchick commented Dec 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants