[presto][iceberg] Add Variant type support for Iceberg V3 (#27374)#27374
Open
apurva-meta wants to merge 7 commits intoprestodb:masterfrom
Open
[presto][iceberg] Add Variant type support for Iceberg V3 (#27374)#27374apurva-meta wants to merge 7 commits intoprestodb:masterfrom
apurva-meta wants to merge 7 commits intoprestodb:masterfrom
Conversation
Contributor
There was a problem hiding this comment.
Sorry @apurva-meta, your pull request is larger than the review limit of 150000 diff characters
|
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
a122607 to
c059615
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
c059615 to
8e88a6d
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
1328b85 to
6c660f1
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
6c660f1 to
4f81b6a
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
4f81b6a to
e48b1c7
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
e48b1c7 to
aa46503
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
aa46503 to
06a820b
Compare
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
06a820b to
4e07636
Compare
added 3 commits
March 19, 2026 14:56
…tensibility Summary: - Reformat FileContent enum in presto_protocol_iceberg.h from single-line to multi-line for better readability and future extension. - Add blank line for visual separation before infoColumns initialization. Protocol files are auto-generated from Java sources via chevron. The manual edits here mirror what the generator would produce once the Java changes are landed and the protocol is regenerated. Differential Revision: D95597575
…equality delete conflict resolution Differential Revision: D97157318
…discovery Summary: Iceberg V3 introduces deletion vectors stored as blobs inside Puffin files. Previously, the coordinator's IcebergSplitSource rejected PUFFIN-format delete files with a NOT_SUPPORTED error, preventing V3 deletion vectors from being discovered and sent to workers. This diff: 1. Adds PUFFIN to the FileFormat enum (both presto-trunk and presto-facebook-trunk) so fromIcebergFileFormat() can convert Iceberg's PUFFIN format to Presto's FileFormat.PUFFIN. 2. Removes the PUFFIN rejection check in presto-trunk's IcebergSplitSource.toIcebergSplit(), allowing deletion vector files to flow through to workers. 3. Updates TestIcebergV3 to verify PUFFIN files are accepted rather than rejected at split enumeration time. The C++ worker-side changes (protocol enum + connector conversion) will follow in a separate diff. Differential Revision: D96666601
added 3 commits
March 19, 2026 14:56
…nd connector layer
Summary:
This is the C++ counterpart to the Java PUFFIN support diff. It wires
the PUFFIN file format through the Prestissimo protocol and connector
conversion layer so that Iceberg V3 deletion vector files can be
deserialized and handled by native workers.
Changes:
1. Adds PUFFIN to the C++ protocol FileFormat enum and its JSON
serialization table in presto_protocol_iceberg.{h,cpp}.
2. Handles PUFFIN in toVeloxFileFormat() in
IcebergPrestoToVeloxConnector.cpp, mapping it to DWRF as a
placeholder since DeletionVectorReader reads raw binary and
does not use the DWRF/Parquet reader infrastructure.
Differential Revision: D96666600
…age sink, compaction procedure, and C++ writer (prestodb#27350) Summary: Add the complete Iceberg V3 deletion vector write path to Presto, including a DV page sink, a DV compaction stored procedure, and a C++ roaring bitmap writer with tests. **Java — DV write path:** - IcebergDeletionVectorPageSink: New ConnectorPageSink that collects deleted row positions, serializes them as a roaring bitmap (cookie=12346 portable format), and writes a Puffin file containing the deletion vector blob. Used by IcebergUpdateablePageSource for V3 tables (format_version >= 3). - RewriteDeleteFilesProcedure: New stored procedure for DV compaction. Groups DVs by data file, detects files with 2+ DVs, deserializes roaring bitmaps, unions the position sets, writes consolidated Puffin files, and commits via RewriteFiles transaction. Usage: CALL iceberg.system.rewrite_delete_files('schema', 'table') - CommitTaskData: Added contentOffset, contentSizeInBytes, recordCount fields for DV metadata. Added backward-compatible constructor overload. - IcebergAbstractMetadata: PUFFIN format handling in delete file commit path using withContentOffset/withContentSizeInBytes/withRecordCount. Removed rejection of V3 column default values (now supported). Added withReferencedDataFile support. - IcebergCommonModule: Registered RewriteDeleteFilesProcedure in Guice bindings. - IcebergPageSourceProvider: Routes V3 tables (format_version >= 3) to IcebergDeletionVectorPageSink; V2 tables continue using IcebergDeletePageSink. - IcebergUpdateablePageSource: Changed delete sink type from IcebergDeletePageSink to ConnectorPageSink interface to support both V2 and V3 delete sinks. - IcebergUtil: Raised MAX_FORMAT_VERSION_FOR_ROW_LEVEL_OPERATIONS from 2 to 3, enabling row-level DELETE/UPDATE on V3 tables. **C++ — Velox DeletionVectorWriter:** - DeletionVectorWriter.h/cpp: C++ roaring bitmap serializer for deletion vectors. Supports array, bitmap, and run containers with offset sections for 4+ containers. Produces Iceberg-compatible portable format (cookie=12346). - DeletionVectorWriterTest.cpp: 8 unit tests covering single position, multiple positions, empty input, consecutive positions, large cardinality (bitmap containers), run containers, and 4+ containers with offsets. **Test:** - TestIcebergV3: Added testDeletionVectorEndToEnd validating the complete coordinator-side DV lifecycle, and testRewriteDeleteFilesProcedure validating DV compaction. Pull Request resolved: prestodb#27350 Differential Revision: D96668349
…ort for Iceberg V3 Summary: Iceberg V3 introduces nanosecond-precision timestamp types (timestamp_ns and timestamptz_ns). This diff adds support for reading tables with these column types by mapping them to Presto's best available precision (TIMESTAMP_MICROSECONDS for timestamp_ns, TIMESTAMP_WITH_TIME_ZONE for timestamptz_ns). Changes: - TypeConverter: Map TIMESTAMP_NANO to Presto types and ORC types - ExpressionConverter: Fix predicate pushdown for TIMESTAMP_MICROSECONDS precision (was incorrectly converting microseconds as milliseconds) - IcebergUtil: Handle TIMESTAMP_NANO partition values (nanos → micros) - PartitionData: Handle TIMESTAMP_NANO in JSON partition deserialization - PartitionTable: Convert nanosecond partition values to microseconds - TestIcebergV3: Add testNanosecondTimestampSchema integration test Differential Revision: D96753790
apurva-meta
added a commit
to apurva-meta/presto
that referenced
this pull request
Mar 19, 2026
…todb#27374) Summary: Pull Request resolved: prestodb#27374 == RELEASE NOTES == General Changes * Upgrade Apache Iceberg library from 1.10.0 to 1.10.1. Hive Connector Changes * Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring bitmaps, including a DV reader, writer, page sink, and compaction procedure. * Add Iceberg equality delete file reader with sequence number conflict resolution per the Iceberg V2+ spec: equality deletes skip when deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips. * Wire dataSequenceNumber through the Presto protocol layer (Java → C++) to enable server-side sequence number conflict resolution for all delete file types. * Add PUFFIN file format support for deletion vector discovery, enabling the coordinator to locate DV files during split creation. * Add Iceberg V3 deletion vector write path with DV page sink and rewrite_delete_files compaction procedure for DV maintenance. * Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3 tables. * Add Variant type support for Iceberg V3, enabling semi-structured data columns in Iceberg tables. * Eagerly collect delete files during split creation with improved logging for easier debugging of Iceberg delete file resolution. * Improve IcebergSplitReader error handling and fix test file handle leaks. * Add end-to-end integration tests for Iceberg V3 covering snapshot lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE, MERGE, time-travel) and all 99 TPC-DS queries. Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - TestIcebergV3: Add comprehensive e2e tests for VARIANT type The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or NestedType), so ColumnIdentity handling works automatically via the existing !isNestedType() check. Differential Revision: D96755027
4e07636 to
283f6ec
Compare
…7374) Summary: Pull Request resolved: prestodb#27374 Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type conversion pipeline, including a binary codec for the Apache Variant spec and SQL scalar functions for Variant data manipulation. The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's unbounded VARCHAR type. This diff provides: 1. Type mapping: VARIANT to/from VARCHAR across all converter layers 2. VariantBinaryCodec: Full encoder/decoder for Apache Variant binary format - Supports primitives (null, bool, int8/16/32/64, double, string) - Supports short strings (0-63 bytes) and long strings - Supports objects (with metadata dictionary) and arrays - JSON string round-trip: JSON -> binary -> JSON - Binary format detection (isVariantBinary) and auto-decode (decodeVariantAuto) - Type introspection from binary headers (getValueTypeName) 3. SQL scalar functions registered via IcebergConnector.getSystemFunctions(): - variant_get(varchar, varchar): Extract field with dot-path and array indexing (e.g., 'users[0].name', 'address.city') - variant_keys(varchar): Return top-level object keys as JSON array - variant_type(varchar): Return JSON type name (object, array, string, number, boolean, null) - to_variant(varchar): Validate JSON and cast to Variant (Phase 5 CAST support) - parse_variant(varchar): Validate and normalize through Variant binary codec - variant_to_json(varchar): Normalize Variant to compact JSON representation - variant_binary_roundtrip(varchar): Encode to binary and decode back (interop testing) 4. Predicate pushdown: IS NULL/IS NOT NULL works through VARCHAR mapping; variant_get pushdown tracked as future optimizer rule work 5. Comprehensive tests for the codec, functions, and end-to-end connector behavior Changes: - TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to ORC STRING type in toOrcType() - IcebergUtil: Handle VARIANT partition values as string slices in domain creation - PartitionData: Deserialize VARIANT partition values as text (same as STRING) - PartitionTable: Convert VariantType partition values to string representation - VariantBinaryCodec: Full Apache Variant binary spec (v1) encoder/decoder with binary detection, type introspection, and auto-decode capabilities - VariantFunctions: 7 SQL scalar functions for Variant data manipulation including dot-path navigation, array indexing, key enumeration, type introspection, CAST - IcebergConnector: Register VariantFunctions in getSystemFunctions() - TestVariantBinaryCodec: 40+ tests covering primitives, objects, arrays, metadata, binary detection, type names, and auto-decode - TestVariantFunctions: 40+ tests covering all 7 functions including dot-path, array indexing, error handling, and edge cases - TestIcebergV3: Integration tests for VARIANT type including JSON data round-trip Note: Velox/Prestissimo does not require changes - the VARIANT->VARCHAR type mapping flows automatically through HiveTypeParser on the C++ side. Differential Revision: D96755027
283f6ec to
bb241f5
Compare
Contributor
|
Please sign the Presto CLA. Please add a release note - or Please edit the PR title to follow semantic commit style to pass the failing and required CI check. See the failure in the test for advice. If you can't edit the PR title, let us know and we can help. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline, including a binary codec for the Apache Variant spec and
SQL scalar functions for Variant data manipulation.
The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's
unbounded VARCHAR type. This diff provides:
(e.g., 'users[0].name', 'address.city')
variant_get pushdown tracked as future optimizer rule work
Changes:
ORC STRING type in toOrcType()
binary detection, type introspection, and auto-decode capabilities
dot-path navigation, array indexing, key enumeration, type introspection, CAST
binary detection, type names, and auto-decode
array indexing, error handling, and edge cases
Note: Velox/Prestissimo does not require changes - the VARIANT->VARCHAR type
mapping flows automatically through HiveTypeParser on the C++ side.
Differential Revision: D96755027