feat: native chunk query API for allocation-free result reading

[OzeroDev]

Summary

Adds QueryChunksContext, which executes a query and hands DuckDB's native data chunks to a callback, so a caller can read a result without the per-row copy allocations that database/sql performs.

This extension is intended for use cases where the result is only forwarded rather than scanned into Go values (e.g. serialize to NDJSON/CSV and write it to a socket, a file, or another process). On that path every rows.Scan allocation becomes a significant overhead as the value is copied out of the vector, boxed into a driver.Value, copied again into the destination, and then immediately discarded. Reading it through database/sql allocated roughly two objects per row, so a large result turned into millions of short-lived allocations.

There were two pre-existing duckdb-go bugs found and addressed as they directly impacted the mechanics behind chunk query functionality.

Both bug fixes are included in this PR as separate commits:

Join the context interrupter on panic

runWithCtxInterrupt closed its done channel and joined the interrupter goroutine only on the normal return path. If fn panicked, the goroutine survived the call and went on calling duckdb_interrupt on a connection database/sql had already discarded and closed, faulting inside cgo. Found while testing a panicking callback.

Invalidate data chunks on close

close freed the C chunk but left the DataChunk vectors pointing into that memory, so every getter and setter kept reading and writing it.

Motivation

This feature extension came out of adding server-side SQL over stored objects to AIStore (NVIDIA's open-source, MIT-licensed storage stack for AI applications). Currently AIStore returns the full object a client requests but with a duckdb-go integration, a GET can instead carry a SQL statement that DuckDB can 1. validate, 2. run against the locally stored object, and 3. stream the projected/filtered result back to the client.

AIStore needs this path to be both efficient and reliable. This PR addresses both concerns: the chunk API removes the per-row allocations, and two pre-existing duckdb-go bugs that the integration surfaced are fixed here as well. Nothing about the API itself is specific to this use case, it applies to any export or streaming path where rows are forwarded rather than inspected, and both bug fixes are independent of the new API entirely.

Lifetime and ownership

The library owns every C allocation involved. The result is destroyed on return, each chunk is freed once consume has seen it, and the prepared statement is closed even if consume panics.

The views alias vector memory and are valid only until the chunk is released (the same behavior as driver.Rows.Next, whose []byte values are only valid until the next call). Because a caller can trivially retain the *DataChunk past its callback, accessing a released chunk now returns errClosedChunk rather than reading freed memory.

API Usage

err := duckdb.QueryChunksContext(ctx, conn, query, func(chunk *duckdb.DataChunk) error {
    for row := range chunk.GetSize() {
        value, notNull, err := chunk.GetVarcharView(0, row)
        if err != nil {
            return err
        }
        if notNull {
            buf = append(buf, value...)
        }
    }
    return nil
})

• QueryChunksContext(ctx, *sql.Conn, query, consume, args...): invokes the consume function on every chunk until all query result chunks have been consumed. Returning an error from consume stops iteration and returns that error unchanged, so a sentinel can stop early.
• DataChunk.GetVarcharView / GetBlobView: the bytes where DuckDB already holds them.
• DataChunk.ColumnNames / ColumnName: the names of columns in DataChunk.
• args: converted the same way database/sql converts them, so a query accepts the same types on both paths.

See chunk_query_bench_test.go file for usage example.

Benchmark

This PR also creates a chunk_query_bench_test.go file to benchmark the performance improvement that the native chunk query API provides over the database/sql usage.

The benchmark serializes 100k rows to a single VARCHAR column and appends it into a reusable
buffer (BenchmarkQueryChunks vs BenchmarkQueryRows):

	ns/op	B/op	allocs/op
QueryChunksContext + views	13,216,832	10,952	266
database/sql rows + Scan	16,206,268	6,410,840	200,226

~750x fewer allocations, ~580x less garbage, ~19% less wall time.

Commits

Each builds and passes tests on its own:

1. Join the context interrupter on panic
2. Extract a borrowed-bytes vector getter
3. Invalidate data chunks on close
4. Add data chunk column name accessors
5. Add borrowed views for VARCHAR and BLOB values
6. Add QueryChunksContext

Commits 1 & 3 are duckdb-go fixes that affect chunk query implementation.
Commits 2, 4 & 5 provide helpers necessary for chunk query implementation.
Commit 6 introduces the core native chunk query API with full test suite and benchmark.

Test plan

• Iteration: TestQueryChunksContext (multi-chunk result, every row seen exactly once), TestQueryChunksContextMultipleStmts (only the last statement yields chunks), TestQueryChunksContextNoRows (consume not called for an empty result), TestQueryChunksContextArgs (positional and sql.Named)
• Value views: TestQueryChunksContextViews (NULL vs. empty string, type mismatch, out-of-range row), TestQueryChunksContextInlineBoundary (lengths 1/11/12/13/24, i.e. both sides of the string_t inline length where decoding switches from inlined bytes to an out-of-line pointer)
• Column metadata: TestQueryChunksContextColumnNames
• Chunk lifetime: TestQueryChunksContextAfterClose: a retained chunk returns errClosedChunk from every getter and setter instead of reading freed memory
• Cancellation: TestQueryChunksContextCancel (stops mid-stream, the connection stays usable afterwards), TestQueryChunksContextCanceledBeforeCall (consume never runs)
• Errors and panics: TestQueryChunksContextQueryError, TestQueryChunksContextConsumerError (sentinel returned unchanged), TestQueryChunksContextConsumerPanic, TestQueryChunksContextRejectsNilConsumer, TestQueryChunksContextRejectsNilConnection, TestRunWithCtxInterrupt_PanicJoinsInterrupter
• No leaked C memory: under -tags=debug_bindings the bindings' allocation counters are zero after each of: happy path, consumer error mid-stream, query error, unsupported column type, panicking consumer
• Full suite green plain, under -race, and under -tags=debug_bindings

[EtgarDev]

makes sense! thank you for the fix :)

[OzeroDev]

This PR is currently marked as a draft as from our testing, we have found that the peak RSS grows in line with the size of the query result. Such behavior is not acceptable for our use case given the size of the objects that we work with, but this problem can be mitigated with streaming (PendingPreparedStreaming). The following commit builds on top of the current PR to stream the chunk queries, though unfortunately, it uses the PendingPreparedStreaming mapping from the core DuckDB library which has been marked as deprecated. I have reached out to the DuckDB team via email regarding the deprecation status but have not heard back from them.

I have also taken a look at the scalar UDF chunk API, but it is invoked concurrently from all worker threads so chunks arrive out of order, and writing to the client from inside the callback blocks DuckDB's worker threads on consumer I/O. Furthermore, the scalar UDF chunk API would also require further work to reduce its GO heap allocation for allocation-free result reading.

Given such circumstance, what does the best course of action look like for duckdb-go and our integration? The streaming chunks approach might provide the most optimized implementation but I am not sure how feasible it is to get PendingPreparedStreaming out of deprecated status.
@EtgarDev

[taniabogatsch]

Hi @OzeroDev - I expect to discuss these changes and the path forward more with @EtgarDev tomorrow/this week, but I already wanted to get back to you on some of your points.

• Indeed, the current streaming API is deprecated. In the upcoming v2.0 release this fall, however, we'll also release the C API V2 (cc @Maxxen), which, on default, has a much more matured streaming result API. We've started to roll out that API on duckdb/duckdb main here, and once more PRs on the actual new interface have landed, duckdb-go will start having a preview branch.
• As part of that preview branch (but also already in this PR), we can start sketching out how a streaming result API in duckdb-go would have to look like (outside of the database/sql interface).

As you mentioned, there is also adjacent work here.

1. Allocations (which you're addressing already with views)
2. Vectorization, or rather, that duckdb-go currently has mostly row-based APIs.

[davidpang731]

Thanks for working on this. We were very glad to see that this PR has arrived at a similar approach to the one we have been exploring independently for our downstream use case.

We previously used duckdb-go through database/sql, but moved to duckdb-go-bindings directly and implemented our own connection pool and borrowed-chunk reader specifically to avoid the per-row memory copies and allocations in the row scanning path.

We were very glad to see this PR arrive at a similar native-chunk approach while preserving the duckdb-go and database/sql integration. A supported API like this could allow us to move back to duckdb-go and remove a substantial amount of locally maintained bindings, pooling, cancellation, and result-lifetime code.

Reducing memory copies is important for our large Parquet scans. Bounded RSS and true streaming are equally important, so we would be particularly interested in the C API V2 streaming version. We’d be happy to test it against our workload and share benchmark results if useful.