Skip to content

feat: Staging table support for MERGE-based updates#243

Merged
phillipleblanc merged 1 commit intotrunkfrom
phillip/260403-bulk-ingest-create-like-table
Apr 6, 2026
Merged

feat: Staging table support for MERGE-based updates#243
phillipleblanc merged 1 commit intotrunkfrom
phillip/260403-bulk-ingest-create-like-table

Conversation

@phillipleblanc
Copy link
Copy Markdown
Contributor

@phillipleblanc phillipleblanc commented Apr 6, 2026

Summary

Add staging table lifecycle support to SpiceBench for MERGE-based updates in the staging_table update strategy.

Changes

system-adapter-protocol:

  • New create_staging_table JSON-RPC method (request/response types, client, server dispatch)
  • Graceful fallback: if the remote adapter returns METHOD_NOT_FOUND, the client logs a warning and treats it as a no-op — newer spicebench works against older adapters

etl/sink/adbc:

  • AdbcSink::new() accepts an optional (Arc<Mutex<SystemAdapterClient>>, Uuid) for remote staging table creation — no separate trait or wrapper needed
  • staging_merge_update() calls the system adapter client directly when provided
  • drop_staging_table_best_effort uses SQL DROP TABLE IF EXISTS (no hook needed)

run.rs:

  • Passes the system adapter client and run ID directly to AdbcSink::new()
  • Strips primary_key_columns from all DatasetConfig when SPICEBENCH_ADBC_UPDATE_STRATEGY=staging_table so tables are created without PKs (avoids Cayenne auto on_conflict: Upsert)

Backward compatibility

The protocol addition is backward compatible: if the adapter doesn't implement create_staging_table, the client catches the METHOD_NOT_FOUND error, logs a warning, and falls back to implicit table creation via bulk ingest.

@phillipleblanc phillipleblanc self-assigned this Apr 6, 2026
@phillipleblanc phillipleblanc force-pushed the phillip/260403-bulk-ingest-create-like-table branch 3 times, most recently from d49be3b to ce1c25e Compare April 6, 2026 06:10
- Add create_staging_table JSON-RPC method to system-adapter-protocol
- AdbcSink accepts optional system adapter client for remote staging table creation
- Strip primary_key_columns when SPICEBENCH_ADBC_UPDATE_STRATEGY=staging_table
- Gracefully fall back if adapter does not support create_staging_table (METHOD_NOT_FOUND)
@phillipleblanc phillipleblanc force-pushed the phillip/260403-bulk-ingest-create-like-table branch from ce1c25e to b715606 Compare April 6, 2026 06:15
@phillipleblanc phillipleblanc merged commit 6327983 into trunk Apr 6, 2026
12 checks passed
@phillipleblanc phillipleblanc deleted the phillip/260403-bulk-ingest-create-like-table branch April 6, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants