Commit c059615
feat: [presto][iceberg] Add Variant type support for Iceberg V3 (prestodb#27374)
Summary:
Pull Request resolved: prestodb#27374
== RELEASE NOTES ==
General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.
Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
resolution per the Iceberg V2+ spec: equality deletes skip when
deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
to enable server-side sequence number conflict resolution for all delete
file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
MERGE, time-travel) and all 99 TPC-DS queries.
Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.
Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type
The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.
Differential Revision: D967550271 parent 02bfd5c commit c059615
File tree
10 files changed
+2565
-1
lines changed- presto-iceberg/src
- main/java/com/facebook/presto/iceberg
- function
- test/java/com/facebook/presto/iceberg
10 files changed
+2565
-1
lines changedLines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
256 | 257 | | |
257 | 258 | | |
258 | 259 | | |
| 260 | + | |
259 | 261 | | |
260 | 262 | | |
261 | 263 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
782 | 782 | | |
783 | 783 | | |
784 | 784 | | |
| 785 | + | |
785 | 786 | | |
786 | 787 | | |
787 | 788 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
| 179 | + | |
179 | 180 | | |
180 | 181 | | |
181 | 182 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
290 | | - | |
| 290 | + | |
291 | 291 | | |
292 | 292 | | |
293 | 293 | | |
| |||
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
| 150 | + | |
| 151 | + | |
150 | 152 | | |
151 | 153 | | |
152 | 154 | | |
| |||
411 | 413 | | |
412 | 414 | | |
413 | 415 | | |
| 416 | + | |
414 | 417 | | |
415 | 418 | | |
416 | 419 | | |
| |||
0 commit comments