fix: support driver manifest use, and adbc_scan batch size hints

Author: rustyconover

CLAUDE.md

@@ -15,16 +15,41 @@ There is also a checkout of the Airport DuckDB extension under ./airport. The Ai
 The extension provides the following functions:
 
 ### Connection Management
-- `adbc_connect(options)` - Connect to an ADBC data source. Returns a connection handle (BIGINT). Options can be passed as a STRUCT (preferred) or MAP. The `driver` option is required.
+- `adbc_connect(options)` - Connect to an ADBC data source. Returns a connection handle (BIGINT). Options can be passed as a STRUCT (preferred) or MAP.
+  - **Required options:**
+    - `driver` - Driver name, path to shared library, or path to manifest file (.toml)
+  - **Optional options:**
+    - `entrypoint` - Custom entry point function name
+    - `search_paths` - Additional paths to search for driver manifests (colon-separated on Unix, semicolon on Windows)
+    - `use_manifests` - Enable/disable manifest search (default: 'true'). Set to 'false' to only use direct library paths.
+    - Other options are passed directly to the ADBC driver
 - `adbc_disconnect(handle)` - Disconnect from an ADBC data source. Returns true on success.
 
+#### Driver Manifest Support
+The extension supports ADBC driver manifests, which allow referencing drivers by name instead of full paths. When `use_manifests` is enabled (default), the driver manager searches for manifests in these locations:
+
+**macOS/Linux:**
+1. `ADBC_DRIVER_PATH` environment variable (colon-separated paths)
+2. `$VIRTUAL_ENV/etc/adbc/drivers` (if in a virtual environment)
+3. `$CONDA_PREFIX/etc/adbc/drivers` (if in a Conda environment)
+4. `~/.config/adbc/drivers` (Linux) or `~/Library/Application Support/ADBC/Drivers` (macOS)
+5. `/etc/adbc/drivers`
+
+**Windows:**
+1. `ADBC_DRIVER_PATH` environment variable (semicolon-separated paths)
+2. Registry: `HKEY_CURRENT_USER\SOFTWARE\ADBC\Drivers\{name}`
+3. `%LOCAL_APPDATA%\ADBC\Drivers`
+4. Registry: `HKEY_LOCAL_MACHINE\SOFTWARE\ADBC\Drivers\{name}`
+
+A manifest file is a TOML file (e.g., `sqlite.toml`) containing driver metadata and the path to the shared library.
+
 ### Transaction Control
 - `adbc_set_autocommit(handle, enabled)` - Enable or disable autocommit mode. When disabled, changes require explicit commit.
 - `adbc_commit(handle)` - Commit the current transaction.
 - `adbc_rollback(handle)` - Rollback the current transaction, discarding all uncommitted changes.
 
 ### Query Execution
-- `adbc_scan(handle, query, [params := row(...)])` - Execute a SELECT query and return results as a table. Supports parameterized queries via the optional `params` named parameter.
+- `adbc_scan(handle, query, [params := row(...)], [batch_size := N])` - Execute a SELECT query and return results as a table. Supports parameterized queries via the optional `params` named parameter. The optional `batch_size` parameter hints to the driver how many rows to return per batch (default: driver-specific, typically 2048). This is a best-effort hint that may be ignored by drivers that don't support it.
 - `adbc_execute(handle, query)` - Execute DDL/DML statements (CREATE, INSERT, UPDATE, DELETE). Returns affected row count.
 - `adbc_insert(handle, table_name, <table>, [mode := ...])` - Bulk insert data from a subquery. Modes: 'create', 'append', 'replace', 'create_append'.
 
@@ -38,15 +63,24 @@ The extension provides the following functions:
 ### Example Usage
 
 ```sql
--- Connect to SQLite (driver path varies by installation)
+-- Connect using a driver manifest (if sqlite.toml is installed in a search path)
+SET VARIABLE conn = (SELECT adbc_connect({'driver': 'sqlite', 'uri': ':memory:'}));
+
+-- Connect with explicit driver path (traditional method)
 SET VARIABLE conn = (SELECT adbc_connect({'driver': '/path/to/libadbc_driver_sqlite.dylib', 'uri': ':memory:'}));
 
+-- Connect with additional search paths
+SET VARIABLE conn = (SELECT adbc_connect({'driver': 'sqlite', 'uri': ':memory:', 'search_paths': '/opt/adbc/drivers'}));
+
 -- Query data
 SELECT * FROM adbc_scan(getvariable('conn')::BIGINT, 'SELECT 1 AS a, 2 AS b');
 
 -- Parameterized query
 SELECT * FROM adbc_scan(getvariable('conn')::BIGINT, 'SELECT ? AS value', params := row(42));
 
+-- Query with batch size hint (for network drivers, larger batches reduce round-trips)
+SELECT * FROM adbc_scan(getvariable('conn')::BIGINT, 'SELECT * FROM large_table', batch_size := 65536);
+
 -- Execute DDL/DML
 SELECT adbc_execute(getvariable('conn')::BIGINT, 'CREATE TABLE test (id INTEGER, name TEXT)');
 SELECT adbc_execute(getvariable('conn')::BIGINT, 'INSERT INTO test VALUES (1, ''hello'')');
@@ -103,10 +137,12 @@ make test
 # Run debug tests
 make test_debug
 
-# Run tests with SQLite driver (requires ADBC_SQLITE_DRIVER env var)
-ADBC_SQLITE_DRIVER="/path/to/libadbc_driver_sqlite.dylib" make test
+# Run tests with SQLite driver (requires both environment variables)
+HAS_ADBC_SQLITE_DRIVER=1 make test
 ```
 
+**Note:** Tests that use the SQLite ADBC driver require the `HAS_ADBC_SQLITE_DRIVER` environment variable to be set (to any value) in addition to `ADBC_SQLITE_DRIVER` pointing to the driver library path.
+
 Tests are written as [SQLLogicTests](https://duckdb.org/dev/sqllogictest/intro.html) in `test/sql/`.
 
 ## Build Outputs

docs/README.md

@@ -408,28 +408,29 @@ Output:
 
 ADBC drivers are available for many databases. Here are some common ones:
 
-| Database | Driver | Notes |
-|----------|--------|-------|
-| SQLite | `libadbc_driver_sqlite.dylib` | Lightweight, file-based |
-| PostgreSQL | `libadbc_driver_postgresql.dylib` | Full-featured RDBMS |
-| Snowflake | `libadbc_driver_snowflake.dylib` | Cloud data warehouse |
-| Flight SQL | `libadbc_driver_flightsql.dylib` | Arrow Flight SQL servers |
-| DuckDB | `libadbc_driver_duckdb.dylib` | Connect to other DuckDB instances |
+• `bigquery` - An ADBC driver for Google BigQuery developed by the ADBC Driver Foundry
+• `duckdb` - An ADBC driver for DuckDB developed by the DuckDB Foundation - **but this would be kind of silly to use in DuckDB**.
+• `flightsql` - An ADBC driver for Apache Arrow Flight SQL developed under the Apache Software Foundation
+• `mssql` - An ADBC driver for Microsoft SQL Server developed by Columnar
+• `mysql` - An ADBC Driver for MySQL developed by the ADBC Driver Foundry
+• `postgresql` - An ADBC driver for PostgreSQL developed under the Apache Software Foundation
+• `redshift` - An ADBC driver for Amazon Redshift developed by Columnar
+• `snowflake` - An ADBC driver for Snowflake developed under the Apache Software Foundation
+• `sqlite` - An ADBC driver for SQLite developed under the Apache Software Foundation
 
 ### Installing Drivers
 
-ADBC drivers can be installed from the [Apache Arrow ADBC releases](https://github.com/apache/arrow-adbc/releases) or built from source.
+There are a few options for installing drivers:
 
-On macOS with Homebrew:
-```bash
-brew install apache-arrow-adbc
-```
+1. [Columnar's `dbc`](https://columnar.tech/dbc/) is a command-line tool that makes installing and managing ADBC drivers easy.
+2. ADBC drivers can be installed from the [Apache Arrow ADBC releases](https://github.com/apache/arrow-adbc/releases) or built from source.
+3. On macOS with Homebrew: ```brew install apache-arrow-adbc```
 
 ## Complete Example
 
 ```sql
 -- Load the extension
-LOAD adbc;
+LOAD adbc_scanner;
 
 -- Connect to SQLite
 SET VARIABLE sqlite_conn = (SELECT adbc_connect({
@@ -502,7 +503,6 @@ SELECT adbc_connect({'uri': ':memory:'});
 - ADBC connections are not automatically closed; always call `adbc_disconnect()` when done
 - The `rows_affected` count from `adbc_execute` depends on driver support; some drivers return 0 for all operations
 - Parameterized queries in `adbc_scan` require the `params` named parameter syntax
-- Connection handles are process-global; be careful with concurrent access from multiple threads
 
 ## Building from Source