Skip to content

[presto][iceberg] Add Variant type support for Iceberg V3 (#27374)#27374

Open
apurva-meta wants to merge 7 commits intoprestodb:masterfrom
apurva-meta:export-D96755027
Open

[presto][iceberg] Add Variant type support for Iceberg V3 (#27374)#27374
apurva-meta wants to merge 7 commits intoprestodb:masterfrom
apurva-meta:export-D96755027

Conversation

@apurva-meta
Copy link

@apurva-meta apurva-meta commented Mar 19, 2026

Summary:

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline, including a binary codec for the Apache Variant spec and
SQL scalar functions for Variant data manipulation.

The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's
unbounded VARCHAR type. This diff provides:

  1. Type mapping: VARIANT to/from VARCHAR across all converter layers
  2. VariantBinaryCodec: Full encoder/decoder for Apache Variant binary format
    • Supports primitives (null, bool, int8/16/32/64, double, string)
    • Supports short strings (0-63 bytes) and long strings
    • Supports objects (with metadata dictionary) and arrays
    • JSON string round-trip: JSON -> binary -> JSON
    • Binary format detection (isVariantBinary) and auto-decode (decodeVariantAuto)
    • Type introspection from binary headers (getValueTypeName)
  3. SQL scalar functions registered via IcebergConnector.getSystemFunctions():
    • variant_get(varchar, varchar): Extract field with dot-path and array indexing
      (e.g., 'users[0].name', 'address.city')
    • variant_keys(varchar): Return top-level object keys as JSON array
    • variant_type(varchar): Return JSON type name (object, array, string, number, boolean, null)
    • to_variant(varchar): Validate JSON and cast to Variant (Phase 5 CAST support)
    • parse_variant(varchar): Validate and normalize through Variant binary codec
    • variant_to_json(varchar): Normalize Variant to compact JSON representation
    • variant_binary_roundtrip(varchar): Encode to binary and decode back (interop testing)
  4. Predicate pushdown: IS NULL/IS NOT NULL works through VARCHAR mapping;
    variant_get pushdown tracked as future optimizer rule work
  5. Comprehensive tests for the codec, functions, and end-to-end connector behavior

Changes:

  • TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
    ORC STRING type in toOrcType()
  • IcebergUtil: Handle VARIANT partition values as string slices in domain creation
  • PartitionData: Deserialize VARIANT partition values as text (same as STRING)
  • PartitionTable: Convert VariantType partition values to string representation
  • VariantBinaryCodec: Full Apache Variant binary spec (v1) encoder/decoder with
    binary detection, type introspection, and auto-decode capabilities
  • VariantFunctions: 7 SQL scalar functions for Variant data manipulation including
    dot-path navigation, array indexing, key enumeration, type introspection, CAST
  • IcebergConnector: Register VariantFunctions in getSystemFunctions()
  • TestVariantBinaryCodec: 40+ tests covering primitives, objects, arrays, metadata,
    binary detection, type names, and auto-decode
  • TestVariantFunctions: 40+ tests covering all 7 functions including dot-path,
    array indexing, error handling, and edge cases
  • TestIcebergV3: Integration tests for VARIANT type including JSON data round-trip

Note: Velox/Prestissimo does not require changes - the VARIANT->VARCHAR type
mapping flows automatically through HiveTypeParser on the C++ side.

Differential Revision: D96755027

@apurva-meta apurva-meta requested review from a team, ZacBlanco and hantangwangd as code owners March 19, 2026 03:39
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @apurva-meta, your pull request is larger than the review limit of 150000 diff characters

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Mar 19, 2026

CLA Missing ID CLA Not Signed

apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
@meta-codesync meta-codesync bot changed the title feat: [presto][iceberg] Add Variant type support for Iceberg V3 feat: [presto][iceberg] Add Variant type support for Iceberg V3 (#27374) Mar 19, 2026
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
@apurva-meta apurva-meta force-pushed the export-D96755027 branch 2 times, most recently from 1328b85 to 6c660f1 Compare March 19, 2026 04:13
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
Apurva Kumar added 3 commits March 19, 2026 14:56
…tensibility

Summary:
- Reformat FileContent enum in presto_protocol_iceberg.h from single-line
  to multi-line for better readability and future extension.
- Add blank line for visual separation before infoColumns initialization.

Protocol files are auto-generated from Java sources via chevron. The manual
edits here mirror what the generator would produce once the Java changes
are landed and the protocol is regenerated.

Differential Revision: D95597575
…equality delete conflict resolution

Differential Revision: D97157318
…discovery

Summary:
Iceberg V3 introduces deletion vectors stored as blobs inside Puffin files.
Previously, the coordinator's IcebergSplitSource rejected PUFFIN-format delete
files with a NOT_SUPPORTED error, preventing V3 deletion vectors from being
discovered and sent to workers.

This diff:
1. Adds PUFFIN to the FileFormat enum (both presto-trunk and
   presto-facebook-trunk) so fromIcebergFileFormat() can convert
   Iceberg's PUFFIN format to Presto's FileFormat.PUFFIN.
2. Removes the PUFFIN rejection check in presto-trunk's
   IcebergSplitSource.toIcebergSplit(), allowing deletion vector
   files to flow through to workers.
3. Updates TestIcebergV3 to verify PUFFIN files are accepted rather
   than rejected at split enumeration time.

The C++ worker-side changes (protocol enum + connector conversion) will
follow in a separate diff.

Differential Revision: D96666601
Apurva Kumar added 3 commits March 19, 2026 14:56
…nd connector layer

Summary:
This is the C++ counterpart to the Java PUFFIN support diff. It wires
the PUFFIN file format through the Prestissimo protocol and connector
conversion layer so that Iceberg V3 deletion vector files can be
deserialized and handled by native workers.

Changes:
1. Adds PUFFIN to the C++ protocol FileFormat enum and its JSON
   serialization table in presto_protocol_iceberg.{h,cpp}.
2. Handles PUFFIN in toVeloxFileFormat() in
   IcebergPrestoToVeloxConnector.cpp, mapping it to DWRF as a
   placeholder since DeletionVectorReader reads raw binary and
   does not use the DWRF/Parquet reader infrastructure.

Differential Revision: D96666600
…age sink, compaction procedure, and C++ writer (prestodb#27350)

Summary:
Add the complete Iceberg V3 deletion vector write path to Presto, including a
DV page sink, a DV compaction stored procedure, and a C++ roaring bitmap
writer with tests.

**Java — DV write path:**

- IcebergDeletionVectorPageSink: New ConnectorPageSink that collects deleted
  row positions, serializes them as a roaring bitmap (cookie=12346 portable
  format), and writes a Puffin file containing the deletion vector blob.
  Used by IcebergUpdateablePageSource for V3 tables (format_version >= 3).
- RewriteDeleteFilesProcedure: New stored procedure for DV compaction. Groups
  DVs by data file, detects files with 2+ DVs, deserializes roaring bitmaps,
  unions the position sets, writes consolidated Puffin files, and commits via
  RewriteFiles transaction.
  Usage: CALL iceberg.system.rewrite_delete_files('schema', 'table')
- CommitTaskData: Added contentOffset, contentSizeInBytes, recordCount fields
  for DV metadata. Added backward-compatible constructor overload.
- IcebergAbstractMetadata: PUFFIN format handling in delete file commit path
  using withContentOffset/withContentSizeInBytes/withRecordCount. Removed
  rejection of V3 column default values (now supported). Added
  withReferencedDataFile support.
- IcebergCommonModule: Registered RewriteDeleteFilesProcedure in Guice bindings.
- IcebergPageSourceProvider: Routes V3 tables (format_version >= 3) to
  IcebergDeletionVectorPageSink; V2 tables continue using IcebergDeletePageSink.
- IcebergUpdateablePageSource: Changed delete sink type from IcebergDeletePageSink
  to ConnectorPageSink interface to support both V2 and V3 delete sinks.
- IcebergUtil: Raised MAX_FORMAT_VERSION_FOR_ROW_LEVEL_OPERATIONS from 2 to 3,
  enabling row-level DELETE/UPDATE on V3 tables.

**C++ — Velox DeletionVectorWriter:**

- DeletionVectorWriter.h/cpp: C++ roaring bitmap serializer for deletion vectors.
  Supports array, bitmap, and run containers with offset sections for 4+
  containers. Produces Iceberg-compatible portable format (cookie=12346).
- DeletionVectorWriterTest.cpp: 8 unit tests covering single position, multiple
  positions, empty input, consecutive positions, large cardinality (bitmap
  containers), run containers, and 4+ containers with offsets.

**Test:**

- TestIcebergV3: Added testDeletionVectorEndToEnd validating the complete
  coordinator-side DV lifecycle, and testRewriteDeleteFilesProcedure validating
  DV compaction.

Pull Request resolved: prestodb#27350

Differential Revision: D96668349
…ort for Iceberg V3

Summary:
Iceberg V3 introduces nanosecond-precision timestamp types (timestamp_ns
and timestamptz_ns). This diff adds support for reading tables with these
column types by mapping them to Presto's best available precision
(TIMESTAMP_MICROSECONDS for timestamp_ns, TIMESTAMP_WITH_TIME_ZONE for
timestamptz_ns).

Changes:
- TypeConverter: Map TIMESTAMP_NANO to Presto types and ORC types
- ExpressionConverter: Fix predicate pushdown for TIMESTAMP_MICROSECONDS
  precision (was incorrectly converting microseconds as milliseconds)
- IcebergUtil: Handle TIMESTAMP_NANO partition values (nanos → micros)
- PartitionData: Handle TIMESTAMP_NANO in JSON partition deserialization
- PartitionTable: Convert nanosecond partition values to microseconds
- TestIcebergV3: Add testNanosecondTimestampSchema integration test

Differential Revision: D96753790
apurva-meta added a commit to apurva-meta/presto that referenced this pull request Mar 19, 2026
…todb#27374)

Summary:
Pull Request resolved: prestodb#27374

== RELEASE NOTES ==

General Changes
* Upgrade Apache Iceberg library from 1.10.0 to 1.10.1.

Hive Connector Changes
* Add Iceberg V3 deletion vector (DV) support using Puffin-encoded roaring
  bitmaps, including a DV reader, writer, page sink, and compaction procedure.
* Add Iceberg equality delete file reader with sequence number conflict
  resolution per the Iceberg V2+ spec: equality deletes skip when
  deleteFileSeqNum <= dataFileSeqNum; positional deletes and DVs skip when
  deleteFileSeqNum < dataFileSeqNum; sequence number 0 (V1 legacy) never skips.
* Wire dataSequenceNumber through the Presto protocol layer (Java → C++)
  to enable server-side sequence number conflict resolution for all delete
  file types.
* Add PUFFIN file format support for deletion vector discovery, enabling
  the coordinator to locate DV files during split creation.
* Add Iceberg V3 deletion vector write path with DV page sink and
  rewrite_delete_files compaction procedure for DV maintenance.
* Add nanosecond timestamp (TIMESTAMP_NANO) type support for Iceberg V3
  tables.
* Add Variant type support for Iceberg V3, enabling semi-structured data
  columns in Iceberg tables.
* Eagerly collect delete files during split creation with improved logging
  for easier debugging of Iceberg delete file resolution.
* Improve IcebergSplitReader error handling and fix test file handle leaks.
* Add end-to-end integration tests for Iceberg V3 covering snapshot
  lifecycle (INSERT, DELETE with equality/positional/DV deletes, UPDATE,
  MERGE, time-travel) and all 99 TPC-DS queries.

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline. The VARIANT type represents semi-structured data (JSON) and is
mapped to Presto's unbounded VARCHAR type.

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- TestIcebergV3: Add comprehensive e2e tests for VARIANT type

The VARIANT type in Iceberg 1.10.0 implements Type directly (not PrimitiveType or
NestedType), so ColumnIdentity handling works automatically via the existing
!isNestedType() check.

Differential Revision: D96755027
…7374)

Summary:
Pull Request resolved: prestodb#27374

Add support for Iceberg V3 VARIANT type across the Presto-Iceberg connector type
conversion pipeline, including a binary codec for the Apache Variant spec and
SQL scalar functions for Variant data manipulation.

The VARIANT type represents semi-structured data (JSON) and is mapped to Presto's
unbounded VARCHAR type. This diff provides:

1. Type mapping: VARIANT to/from VARCHAR across all converter layers
2. VariantBinaryCodec: Full encoder/decoder for Apache Variant binary format
   - Supports primitives (null, bool, int8/16/32/64, double, string)
   - Supports short strings (0-63 bytes) and long strings
   - Supports objects (with metadata dictionary) and arrays
   - JSON string round-trip: JSON -> binary -> JSON
   - Binary format detection (isVariantBinary) and auto-decode (decodeVariantAuto)
   - Type introspection from binary headers (getValueTypeName)
3. SQL scalar functions registered via IcebergConnector.getSystemFunctions():
   - variant_get(varchar, varchar): Extract field with dot-path and array indexing
     (e.g., 'users[0].name', 'address.city')
   - variant_keys(varchar): Return top-level object keys as JSON array
   - variant_type(varchar): Return JSON type name (object, array, string, number, boolean, null)
   - to_variant(varchar): Validate JSON and cast to Variant (Phase 5 CAST support)
   - parse_variant(varchar): Validate and normalize through Variant binary codec
   - variant_to_json(varchar): Normalize Variant to compact JSON representation
   - variant_binary_roundtrip(varchar): Encode to binary and decode back (interop testing)
4. Predicate pushdown: IS NULL/IS NOT NULL works through VARCHAR mapping;
   variant_get pushdown tracked as future optimizer rule work
5. Comprehensive tests for the codec, functions, and end-to-end connector behavior

Changes:
- TypeConverter: Map VARIANT to VarcharType (unbounded) in toPrestoType(), and to
  ORC STRING type in toOrcType()
- IcebergUtil: Handle VARIANT partition values as string slices in domain creation
- PartitionData: Deserialize VARIANT partition values as text (same as STRING)
- PartitionTable: Convert VariantType partition values to string representation
- VariantBinaryCodec: Full Apache Variant binary spec (v1) encoder/decoder with
  binary detection, type introspection, and auto-decode capabilities
- VariantFunctions: 7 SQL scalar functions for Variant data manipulation including
  dot-path navigation, array indexing, key enumeration, type introspection, CAST
- IcebergConnector: Register VariantFunctions in getSystemFunctions()
- TestVariantBinaryCodec: 40+ tests covering primitives, objects, arrays, metadata,
  binary detection, type names, and auto-decode
- TestVariantFunctions: 40+ tests covering all 7 functions including dot-path,
  array indexing, error handling, and edge cases
- TestIcebergV3: Integration tests for VARIANT type including JSON data round-trip

Note: Velox/Prestissimo does not require changes - the VARIANT->VARCHAR type
mapping flows automatically through HiveTypeParser on the C++ side.

Differential Revision: D96755027
@meta-codesync meta-codesync bot changed the title feat: [presto][iceberg] Add Variant type support for Iceberg V3 (#27374) [presto][iceberg] Add Variant type support for Iceberg V3 (#27374) Mar 19, 2026
@steveburnett
Copy link
Contributor

Please sign the Presto CLA.

Please add a release note - or NO RELEASE NOTE - following the Release Notes Guidelines to pass the failing but not required CI check.

Please edit the PR title to follow semantic commit style to pass the failing and required CI check. See the failure in the test for advice. If you can't edit the PR title, let us know and we can help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants