duckdb extension: crash when array field is NULL

[uwemaurer]

Issue Description

$ RUST_BACKTRACE=1 duckdb
DuckDB v1.4.2 (Andium) 68d7555f68
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D load vortex;
D update extensions;
┌────────────────┬────────────┬─────────────────────┬──────────────────┬─────────────────┐
│ extension_name │ repository │    update_result    │ previous_version │ current_version │
│    varchar     │  varchar   │       varchar       │     varchar      │     varchar     │
├────────────────┼────────────┼─────────────────────┼──────────────────┼─────────────────┤
│ sqlite_scanner │ core       │ NO_UPDATE_AVAILABLE │ 0c93d61          │ 0c93d61         │
│ vortex         │ core       │ NO_UPDATE_AVAILABLE │ dae36cd          │ dae36cd         │
└────────────────┴────────────┴─────────────────────┴──────────────────┴─────────────────┘
D select * from values ('1',['test']), ('2', null) as v(x,y);
┌─────────┬───────────┐
│    x    │     y     │
│ varchar │ varchar[] │
├─────────┼───────────┤
│ 1       │ [test]    │
│ 2       │ NULL      │
└─────────┴───────────┘
D copy (select * from values ('1',['test']), ('2', null) as v(x,y)) to 'a.vortex' (format vortex);
Executor Error:
views to the end of the elements array (length 0) must have size 0
Backtrace:
   0: vortex_array::arrays::listview::array::validate_offsets_and_sizes
   1: vortex_array::arrays::listview::array::ListViewArray::validate
   2: vortex_array::arrays::listview::array::ListViewArray::try_new
   3: vortex_duckdb::convert::vector::flat_vector_to_vortex
   4: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
   5: vortex_duckdb::duckdb::copy_function::callback::copy_to_sink_callback
   6: _ZN6vortex14c_copy_to_sinkERN6duckdb16ExecutionContextERNS0_12FunctionDataERNS0_18GlobalFunctionDataERNS0_17LocalFunctionDataERNS0_9DataChunkE
   7: _ZNK6duckdb18PhysicalCopyToFile4SinkERNS_16ExecutionContextERNS_9DataChunkERNS_17OperatorSinkInputE
   8: _ZN6duckdb16PipelineExecutor19ExecutePushInternalERNS_9DataChunkERNS_15ExecutionBudgetEm
   9: _ZN6duckdb16PipelineExecutor7ExecuteEm
  10: _ZN6duckdb12PipelineTask11ExecuteTaskENS_17TaskExecutionModeE
  11: _ZN6duckdb12ExecutorTask7ExecuteENS_17TaskExecutionModeE
  12: _ZN6duckdb8Executor11ExecuteTaskEb
  13: _ZN6duckdb13ClientContext19ExecuteTaskInternalERNS_17ClientContextLockERNS_15BaseQueryResultEb
  14: _ZN6duckdb18PendingQueryResult15ExecuteInternalERNS_17ClientContextLockE
  15: _ZN6duckdb18PendingQueryResult7ExecuteEv
  16: duckdb_shell_sqlite3_print_duckbox
  17: _ZN12duckdb_shell10ShellState24ExecutePreparedStatementEP12sqlite3_stmt
  18: _ZN12duckdb_shell10ShellState10ExecuteSQLEPKcPPc
  19: _ZN12duckdb_shell10ShellState13RunOneSqlLineENS_9InputModeEPc
  20: _ZN12duckdb_shell10ShellState12ProcessInputENS_9InputModeE
  21: main
  22: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  23: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  24: _start

D 

Expected Behavior

correctly handle the NULL array field

Actual Behavior

fails to handle the array field when it is null

Reproduction Steps

see detailed steps above

OS Version Information

ubuntu 24

I acknowledge that:

• I have searched the Vortex repository (both open and closed Discussions and Issues) and confirm this is not a duplicate of an existing issue or discussion.
• I have checked the "Preview" tab on all text fields to ensure that everything looks right, and have wrapped all configuration and code in code blocks with a group of three backticks (```) on separate lines.

[joseph-isaacs]

@connortsui20 we don't check the validity when exporting to duckdb

vortex/vortex-duckdb/src/exporter/varbinview.rs

Line 63 in ba3b2e5

for (mut_view, view) in unsafe { vector.as_slice_mut::<PtrBinaryView>(len) }

[connortsui20]

I think this might play a part in it but there also might be an issue with how we convert duckdb lists to our list views

[joseph-isaacs]

You are correct

[connortsui20]

Yeah I think something is wrong there:

D copy (select * from values ('1',['test']), ('2', null) as v(x,y)) to 'a.vortex' (format vortex);
[vortex-duckdb/src/convert/vector.rs:246:13] &offsets = Buffer<i64> {
    length: 2,
    alignment: Alignment(
        8,
    ),
    as_slice: [0, 0],
}
[vortex-duckdb/src/convert/vector.rs:247:13] &lengths = Buffer<i64> {
    length: 2,
    alignment: Alignment(
        8,
    ),
    as_slice: [1, 0],
}
Executor Error:
views to the end of the elements array (length 0) must have size 0 (had size 1)

[connortsui20]

Ah I found the issue:

            let child_data = flat_vector_to_vortex(
                &mut vector.list_vector_get_child(),
                usize::try_from(offsets[len - 1] + lengths[len - 1])
                    .vortex_expect("last offset and length sum must fit in usize "),
            )?;

We are not guaranteed that the last offset + length actually represents the length of the vector/array since views do not have to be in order.

Will fix this soon.

[connortsui20]

#5740 will fix this!

Apologies for the late response, we will do our best to be more on top of issues in the future.

[uwemaurer]

There is a great way to test all these data types from duckdb: https://duckdb.org/docs/stable/dev/sqllogictest/writing_tests#functions-for-generating-test-data

You can do

 copy (select * from test_all_types()) to 'a.vortex' (format vortex);

(or use select * exclude (...) to remove the unsupported ones)