Skip to content

Unify vector setter dispatch - #164

Draft
mlafeldt wants to merge 4 commits into
duckdb:mainfrom
mlafeldt:unify-vector-setter-dispatch
Draft

Unify vector setter dispatch#164
mlafeldt wants to merge 4 commits into
duckdb:mainfrom
mlafeldt:unify-vector-setter-dispatch

Conversation

@mlafeldt

@mlafeldt mlafeldt commented Jul 24, 2026

Copy link
Copy Markdown
Member

Use the setter selected during vector initialization as the single correctness dispatch for chunk, row, appender, and table UDF writes.

  • Adds ARRAY support to SetChunkValue.
  • Writes nil values, including typed-nil *big.Int, as SQL NULL.
  • Retains typed and numeric fast paths; uncommon fallback conversions may box.

BREAKING: JSON writes now match SetValue and appenders. Strings become JSON strings, []byte becomes a base64-encoded JSON string, and maps or structs become JSON objects. Use json.RawMessage for pre-serialized JSON documents.


Go input Current main After this PR
string("hello") Writes invalid raw JSON hello; reading returns an API error wrapping json.SyntaxError Writes "hello"; reads back as Go string "hello"
string("{\"a\":1}") Writes raw object {"a":1}; reads back as a map Writes "{\"a\":1}"; reads back as Go string {"a":1}
[]byte("{\"a\":1}") Writes raw object {"a":1}; reads back as a map Writes "eyJhIjoxfQ=="; reads back as Go string "eyJhIjoxfQ=="
map[string]any{"a": 1} Cast error Writes {"a":1}; reads back as a map
json.RawMessage("{\"a\":1}") Cast error Writes raw object {"a":1}; reads back as a map

@mlafeldt
mlafeldt requested a review from taniabogatsch July 24, 2026 13:30
@mlafeldt

Copy link
Copy Markdown
Member Author

@taniabogatsch I think this breaking change is justified. Agree?

Route every vector write through the setter selected during vector
initialization. Remove the duplicate type switch and its identity wrapper, and
drop generic parameters from setters that now always receive any values.

This adds ARRAY support to SetChunkValue and makes nil values set SQL NULL
instead of panicking.

BREAKING: SetChunkValue and SetRowValue no longer treat strings as raw JSON
documents. Strings are marshaled as JSON string values, maps and structs as
JSON objects, and pre-serialized documents should use json.RawMessage. This
matches DataChunk.SetValue and appender behavior.
@mlafeldt
mlafeldt force-pushed the unify-vector-setter-dispatch branch from 9a0d3a4 to 406940d Compare July 24, 2026 13:34
@taniabogatsch

Copy link
Copy Markdown
Member

The breaking change is fine by me.

Another question, I remember the PR that added the setVectorValue interface, maybe worth checking that we don't regress here? marcboeker#276

@mlafeldt
mlafeldt force-pushed the unify-vector-setter-dispatch branch from 406940d to 7d1b75b Compare July 24, 2026 15:33
marcboeker#276 made SetChunkValue
generic to avoid allocating when table UDFs write values. Routing every
write through the any-valued callback restored correctness locality but
boxed each generic value.

Install optional exact-type callbacks for each vector canonical Go type,
with setFn as the fallback for all other values, and keep numeric
conversion setters generic. The numeric optimization defaults to the
vector-installed setFn, so types it does not recognize retain the single
correctness dispatch in vector initialization.

Add benchmarks for both public table UDF write paths and a matrix
covering cross-numeric and representative canonical writes. These paths
now match or improve on main allocation counts.
@mlafeldt
mlafeldt force-pushed the unify-vector-setter-dispatch branch from 7d1b75b to f877027 Compare July 24, 2026 16:11
mlafeldt added 2 commits July 24, 2026 18:33
Let HUGEINT, UHUGEINT, and BIGNUM writes use the vector-installed
setter instead of the numeric fast path. Their setters normalize typed-nil
*big.Int values to SQL NULL, so intercepting them changed SetChunkValue
semantics and violated the fallback invariant.

Document that exact typed setters bypass nil normalization and that the
numeric optimization only covers setters with an interface-nil guard.
Clarify how SetChunkValue can avoid boxing exact canonical values, and align
ENUM initialization with the other typed setters.
Move the generic SetChunkValue benchmark helper and type matrix into one
file so the exact BIGINT case does not duplicate setup.

Cover []byte-to-VARCHAR and string-to-BLOB fallback writes explicitly.
Canonical fast paths preserve their allocation behavior, while supported
cross-type fallback writes may box. The benchmark suite now exposes that
boundary instead of implying every accepted conversion is allocation-free.
@mlafeldt
mlafeldt marked this pull request as draft July 24, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants