[Iceberg v3] Row lineage by dain · Pull Request #27836 · trinodb/trino

dain · 2026-01-03T03:58:23Z

Description

This PR adds comprehensive support for Iceberg v3 row lineage, enabling Trino to read and preserve row identity metadata ($row_id and $last_updated_sequence_number).

Changes include:

Reading lineage columns: Expose $row_id and $last_updated_sequence_number as queryable columns for v3 tables
OPTIMIZE is now enabled for v3 tables and preserves lineage data
UPDATE and MERGE operations preserve the original $row_id for modified rows, maintaining row identity across updates
Enable cleanup procedures (expire_snapshots, remove_orphan_files) on v3 tables
Run Iceberg connector tests against both v2 and v3 format versions

Release notes

(X) Release notes are required, with the following suggested text:

## Iceberg
* Add support for Iceberg v3 row lineage, including reading `$row_id` and `$last_updated_sequence_number` columns, preserving row identity during UPDATE/MERGE operations, and OPTIMIZE support. ({issue}`issuenumber)

chenjian2664 · 2026-01-06T04:27:00Z

Is the second commit "add support for Iceberg v3 deletion vectors" added intentionally?

dain · 2026-01-09T07:47:58Z

Is the second commit "add support for Iceberg v3 deletion vectors" added intentionally?

Yep. The first two commits are the base from another PR. Row lineage needs deletions to work to fully test that rowid works for update commands.

chenjian2664

Reviewed: "Add support for reading $row_id and $last_updated_sequence_number"

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

chenjian2664 · 2026-01-19T09:35:57Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

@@ -647,6 +647,7 @@ void testV3InsertProducesRowLineageMetadata()
        assertUpdate("CREATE TABLE " + tableName + " (id INTEGER, v VARCHAR) WITH (format = 'PARQUET', format_version = 3)");


ideally, we should also test AVRO and ORC format

I don't think this is necessary. These columns are synthetic and handled by the engine directly. I don't think file format has any impact on this feature.

where is the test case exercise the IcebergPageSourceProvider changes (for AVRO and ORC)

I added that comment after I pushed. I just finally got through applying all of the comments below. This is now covered.

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

chenjian2664 · 2026-01-19T10:00:25Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

+                    if (column.isLastUpdatedSequenceNumberColumn()) {
+                        transforms.transform(new DataSequenceNumberTransform(dataSequenceNumber, ordinal));
+                    }
+                    else if (column.isRowIdColumn() && fileFirstRowId.isPresent()) {
+                        appendRowNumberColumn = true;
+                        transforms.transform(new RowIdTransform(fileFirstRowId.get(), ordinal));


plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

chenjian2664

Reviewed: "Allow table procedures to declare which columns to read"

chenjian2664 · 2026-01-19T10:41:01Z

core/trino-main/src/main/java/io/trino/sql/planner/LogicalPlanner.java

-        Optional<TableLayout> layout = metadata.getLayoutForTableExecute(session, executeHandle);
-
-        List<Symbol> symbols = visibleFields(tableScanPlan);
+        Set<String> expectedColumnNames = metadata.getColumnNamesForTableExecute(session, executeHandle);


What about returns Optional<List<ColumnHandle>> ?

This API is required and null is not an allowed response. The connector metadata has a default implementation so this is backwards compatible.

@Praveen2112 Please take a look

Actually, @electrum pointed out that we could be using ColumHandles here instead of names. I was reacting to the optional part not the column handle part.

chenjian2664 · 2026-01-19T10:41:42Z

core/trino-main/src/main/java/io/trino/metadata/Metadata.java

            String procedureName,
            Map<String, Object> executeProperties);

+    Set<String> getColumnNamesForTableExecute(Session session, TableExecuteHandle tableExecuteHandle);


Involve @Praveen2112 @ebyhr
Please help to review this commit

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergParquetConnectorTest.java

github-actions · 2026-02-10T17:23:04Z

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

electrum · 2026-02-27T17:41:11Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

                else if (!fileColumnsByIcebergId.containsKey(column.getBaseColumnIdentity().getId())) {
-                    Object initialDefault = getInitialDefault(tableSchema, column.getBaseColumnIdentity().getId());
-                    transforms.constantValue(nativeValueToBlock(column.getType(), initialDefault));
+                    if (column.isLastUpdatedSequenceNumberColumn()) {


These should probably be else if on the outer level, after the other if (column.isXxx)

Actually, this is correct. This handles the case where the row id or LUSN is not present in file and we must synthesize them, otherwise we must read them from the file.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMergeSink.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

chenjian2664 · 2026-02-28T10:01:51Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergV3.java

@@ -647,6 +647,7 @@ void testV3InsertProducesRowLineageMetadata()
        assertUpdate("CREATE TABLE " + tableName + " (id INTEGER, v VARCHAR) WITH (format = 'PARQUET', format_version = 3)");


where is the test case exercise the IcebergPageSourceProvider changes (for AVRO and ORC)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMergeSink.java

chenjian2664 · 2026-03-01T07:23:12Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMergeSink.java

+                    }
+                }
+            }
+


add a check to verify that the pendingSourceRowId is null - that we don't have unhandled row ids

I rewrote this

chenjian2664 · 2026-03-01T07:25:31Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMergeSink.java

+        private static Block createRowIdBlock(Page inputPage, int dataColumnCount, int[] additionPositions, int additionCount)
+        {
+            // For V3, we need to extract source_row_id from the merge row ID for UPDATE_INSERT rows.
+            // UPDATE_DELETE is immediately followed by UPDATE_INSERT, so we track pending source row IDs.


UPDATE_DELETE is immediately followed by UPDATE_INSERT, where is the logic that guarantee it?

I rewrote this with more explict handling verification. Generally, none of these update systems actually verify this stuff, but I am happy to add it.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java

ebyhr

I don't think this PR is ready for merge yet. Please request another review round before you merge this PR. The repeated UPDATE scenario is broken:

CREATE TABLE test (name varchar) WITH (format_version = 3);
INSERT INTO test VALUES 'alice', 'bob';
INSERT INTO test VALUES 'carol', 'david';
SELECT name, "$row_id", "$last_updated_sequence_number" FROM test;
 name  | $row_id | $last_updated_sequence_number
-------+---------+-------------------------------
 alice |       0 |                             2
 carol |       2 |                             3
 bob   |       1 |                             2
 david |       3 |                             3

UPDATE test SET name = 'BOB' WHERE name = 'bob';
SELECT name, "$row_id", "$last_updated_sequence_number" FROM test;
 name  | $row_id | $last_updated_sequence_number
-------+---------+-------------------------------
 carol |       2 |                             3
 david |       3 |                             3
 BOB   |       1 |                             4
 alice |       0 |                             2

UPDATE test SET name = 'BOB1' WHERE name = 'BOB';
SELECT name, "$row_id", "$last_updated_sequence_number" FROM test;
 name  | $row_id | $last_updated_sequence_number
-------+---------+-------------------------------
 carol |       2 |                             3
 BOB1  |       4 |                             5
 alice |       0 |                             2
 david |       3 |                             3

The bottom BOB1 row should return 1 on $row_id column.

ebyhr · 2026-03-02T03:39:24Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

    protected TimeUnit storageTimePrecision;

-    protected BaseIcebergConnectorTest(IcebergFileFormat format)
+    protected BaseIcebergConnectorTest(IcebergFileFormat format, int formatVersion)


Please test SHOW STATS with the new metadata columns. I believe it returns incorrect results.

Also, add MV test case:

@Test void testRowLineageWithMaterializedViews() { try (TestTable table = newTrinoTable("test_materialized_views", "(id int, name varchar) WITH (format_version = 3)")) { assertUpdate("INSERT INTO " + table.getName() + " VALUES (1, 'Alice'), (2, 'Bob')", 2); String materializedViewName = "test_materialized_view_" + randomNameSuffix(); assertUpdate("CREATE MATERIALIZED VIEW " + materializedViewName + " AS SELECT id, name, \"$row_id\", \"$last_updated_sequence_number\" FROM " + table.getName()); assertUpdate("REFRESH MATERIALIZED VIEW " + materializedViewName, 2); assertThat(query("SELECT id, name, \"$row_id\", \"$last_updated_sequence_number\" FROM " + materializedViewName)) .matches(""" VALUES (1, VARCHAR 'Alice', BIGINT '0', BIGINT '2'), (2, 'Bob', BIGINT '1', BIGINT '2') """); assertUpdate("UPDATE " + table.getName() + " SET name = 'Alice Updated' WHERE id = 1", 1); assertUpdate("REFRESH MATERIALIZED VIEW " + materializedViewName, 2); assertThat(query("SELECT id, name, \"$row_id\", \"$last_updated_sequence_number\" FROM " + materializedViewName)) .matches(""" VALUES (1, VARCHAR 'Alice Updated', BIGINT '0', BIGINT '3'), (2, 'Bob', BIGINT '1', BIGINT '2') """); assertUpdate("DROP MATERIALIZED VIEW " + materializedViewName); } }

Column name and variable were not updated to be merge specific when the merge PR was being reviewed.

dain · 2026-03-04T06:27:33Z

@ebyhr I believe this is ready to go now

ebyhr · 2026-03-04T07:03:56Z

@dain Thanks for addressing the comments. I'll review this PR again tomorrow or shortly later.

dain · 2026-03-05T19:07:01Z

@ebyhr any updates?

ebyhr · 2026-03-06T00:12:55Z

@dain Sorry, I had to work on a different issue yesterday. It looks like there’s still a bug with the Avro format.

CREATE TABLE test (name varchar) WITH (format_version = 3, format = 'AVRO');
INSERT INTO test VALUES 'alice', 'bob';
INSERT INTO test VALUES 'carol', 'david';
UPDATE test SET name = 'BOB' WHERE name = 'bob';
SELECT name, "$row_id", "$last_updated_sequence_number" FROM test;

 name  | $row_id | $last_updated_sequence_number
-------+---------+-------------------------------
 carol |       2 |                             3
 david |       3 |                             3
 alice |       0 |                             2
 BOB   |       4 |                             4

BOB should return 1 as $row_id.

ebyhr · 2026-03-06T00:42:05Z

The partition table has a bug regardless of the file format:

CREATE TABLE test (name varchar, x bigint) WITH (format_version = 3, partitioning = ARRAY['x']);
INSERT INTO test VALUES ('alice', 1), ('bob', 2);
INSERT INTO test VALUES ('carol', 1), ('david', 2);
UPDATE test SET name = 'BOB' WHERE name = 'bob';
SELECT name, "$row_id" FROM test;

 name  | $row_id
-------+---------
 BOB   |       4
 carol |       2
 david |       3
 alice |       1
(4 rows)

BOB should return 0 as $row_id.

cla-bot bot added the cla-signed label Jan 3, 2026

github-actions bot added iceberg Iceberg connector lakehouse labels Jan 3, 2026

dain force-pushed the row-lineage branch 2 times, most recently from 6782e91 to 39782c8 Compare January 3, 2026 05:38

dain marked this pull request as ready for review January 3, 2026 07:17

dain force-pushed the row-lineage branch from 39782c8 to 85b9c44 Compare January 11, 2026 02:39

dain changed the title ~~Add Iceberg v3 row lineage support~~ [Iceberg v3] Row lineage Jan 11, 2026

dain force-pushed the row-lineage branch from 85b9c44 to b9e2b64 Compare January 16, 2026 07:57

chenjian2664 self-requested a review January 16, 2026 07:58

dain force-pushed the row-lineage branch from b9e2b64 to d1241dd Compare January 16, 2026 08:26

chenjian2664 reviewed Jan 19, 2026

View reviewed changes

raphaelsolarski reviewed Jan 19, 2026

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergParquetConnectorTest.java Show resolved Hide resolved

github-actions bot added the stale label Feb 10, 2026

dain added the stale-ignore Use this label on PRs that should be ignored by the stale bot so they are not flagged or closed. label Feb 17, 2026

dain force-pushed the row-lineage branch 6 times, most recently from f6e342b to 72789e0 Compare February 27, 2026 02:31

electrum approved these changes Feb 28, 2026

View reviewed changes

chenjian2664 reviewed Mar 1, 2026

View reviewed changes

ebyhr reviewed Mar 2, 2026

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergPageSourceProvider.java Show resolved Hide resolved

ebyhr requested changes Mar 2, 2026

View reviewed changes

Fix typo in Iceberg merge row id name

fde4302

Column name and variable were not updated to be merge specific when the merge PR was being reviewed.

dain force-pushed the row-lineage branch from 72789e0 to 66ec210 Compare March 4, 2026 03:45

dain added 6 commits March 3, 2026 20:00

Add support for reading $row_id and $last_updated_sequence_number

6fbef26

Allow table procedures to declare which columns to read

77ceef7

Add support for OPTIMIZE in Iceberg v3

cffb2e7

Preserve row_id in Iceberg v3 updates

037bee0

Allow cleanup procedures on Iceberg v3 tables

1a9c3b2

Run Iceberg connector tests on v2 and v3

d0f51bc

dain force-pushed the row-lineage branch from 66ec210 to d0f51bc Compare March 4, 2026 04:47

		@@ -647,6 +647,7 @@ void testV3InsertProducesRowLineageMetadata()
		assertUpdate("CREATE TABLE " + tableName + " (id INTEGER, v VARCHAR) WITH (format = 'PARQUET', format_version = 3)");

Conversation

dain commented Jan 3, 2026

Description

Release notes

Uh oh!

chenjian2664 commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dain commented Jan 9, 2026

Uh oh!

chenjian2664 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chenjian2664 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjian2664 commented Jan 6, 2026 •

edited

Loading