fix: handle VectorBatchAppender type mismatch in consolidateBatches

[EDsCODE]

Summary

• All production events-ducklake tasks are FAILED with IllegalArgumentException: The targetVector to append must have the same type as the targetVector being appended in consolidateBatches
• Root cause: castBatchToSchema used ArrowType equality for the "same type, direct copy" path. For complex types (Struct, List, Map), ArrowType is always equal (e.g., both ArrowType.Struct) even when children differ. This caused raw copyFromSafe calls between vectors with mismatched child types, which either threw or corrupted data
• Fix 1 (root cause): Changed to Field.equals which compares children, nullability, and metadata. Mismatched fields now go through castVectorValues for proper type promotion
• Fix 2 (missing promotions): Added cast handlers for type pairs that ArrowSchemaMerge.unifySchemas can produce but castVectorValues didn't handle: Bool → Int32/Int64/Float64, TinyInt/SmallInt → Int32
• Fix 3 (safety net): consolidateBatches now catches VectorBatchAppender exceptions and falls back to writing batches individually, instead of crashing the task

Test plan

• SchemaMismatchIntegrationTest — sends records with conflicting types (string vs number) in separate poll cycles, verifies task stays RUNNING after flush
• Unit tests pass
• Deploy to dev and verify tasks recover after restart

🤖 Generated with Claude Code

[EDsCODE]

don't like this. could end up with too many writes

[jghoman]

Rather than falling back on single batches, I've changed the code to write contiguous schema-matching batches in the most efficient way possible. This removed a lot of code, and opened up several other efficiencies. Also added a lot of tests.

BatchConsolidator.java

• Removed unused BufferAllocator allocator parameter from consolidate()
• Removed per-run try/catch that would add corrupt batches to result on partial append failure
• Moved single-batch run check (run.size() == 1) up to the caller loop, removed redundant guard in consolidateRun()
• Rewrote consolidate() javadoc with explicit ownership contract, failure semantics, @param/@return/@throws

DucklakeSinkTask.java (flushBatches)

• Wrapped BatchConsolidator.consolidate() call in its own try/catch that closes all original batches on failure (prevents Arrow memory leak)
• Narrowed write-path catch from Exception to RuntimeException
• Changed throw new RuntimeException("...", e) to throw e (no double-wrap since writer.write() already throws RuntimeException)
• Updated call site to match removed allocator parameter

SchemaMismatchIntegrationTest.java

• Updated class javadoc: "falls back to writing batches individually" → "groups contiguous compatible batches and writes incompatible batches as separate runs"
• Updated test method javadoc: references BatchConsolidator and contiguous-run grouping
• Updated inline comment at the Thread.sleep: "falls back to individual writes" → "writes them as separate runs"

BatchConsolidationTest.java

• Updated all 29 consolidate() call sites to remove allocator argument
• Removed unused assertThrows import (from failed corrupted-vector test exploration)

[jghoman]

This also removes the type promotion that had initially been in the PR.

[Copilot]

Pull request overview

This PR addresses production task failures caused by Arrow VectorBatchAppender type mismatches during batch consolidation by introducing schema-aware consolidation that only appends batches with compatible (including nested-child) schemas, and by moving consolidation logic out of DucklakeSinkTask into a dedicated helper.

Changes:

• Add BatchConsolidator to group contiguous batches by compatible schema and consolidate each run via in-place VectorBatchAppender append.
• Update DucklakeSinkTask.flushBatches to use BatchConsolidator and simplify the flush path by writing one batch per compatible run.
• Add unit + integration coverage for consolidation behavior, schema-compatibility edge cases, and the previously failing mismatch scenario.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/main/java/com/inyo/ducklake/connect/DucklakeSinkTask.java	Switch flush-time consolidation to BatchConsolidator and adjust close/error-handling paths.
src/main/java/com/inyo/ducklake/connect/BatchConsolidator.java	New utility to group by compatible schema and consolidate runs in-place using VectorBatchAppender.
src/test/java/com/inyo/ducklake/connect/BatchConsolidationTest.java	New unit tests for consolidation correctness, schema compatibility, ordering, and memory management.
src/integrationTest/java/com/inyo/ducklake/connect/SchemaMismatchIntegrationTest.java	New integration test reproducing the schema mismatch scenario across poll cycles and asserting the task remains running.
CLAUDE.md	Update local dev environment guidance (mise → flox).