Skip to content

Commit fc09fc7

Browse files
authored
[docs] Add documentation for Flink lineage support (#8476)
1 parent 5c3d4bc commit fc09fc7

2 files changed

Lines changed: 40 additions & 1 deletion

File tree

docs/docs/flink/lineage.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
title: "Data Lineage"
3+
sidebar_position: 13
4+
---
5+
6+
# Data Lineage
7+
8+
Paimon implements [FLIP-314](https://cwiki.apache.org/confluence/spaces/FLINK/pages/255070913/FLIP-314+Support+Customized+Job+Lineage+Listener) to expose lineage information through Flink's native [LineageVertexProvider](https://nightlies.apache.org/flink/flink-docs-stable/docs/internals/data_lineage/) interface. This allows lineage consumers (such as [OpenLineage](https://openlineage.io/docs/integrations/flink/flink2)) to automatically discover Paimon datasets in the Flink job graph.
9+
10+
:::info
11+
Data Lineage support is available only for Flink 2.0 and above.
12+
:::
13+
14+
## Table API / SQL
15+
16+
When using Paimon tables via Flink SQL or Table API, lineage is automatically reported for both source and sink tables.
17+
18+
Apart from the table lineage information provided by Flink natively (name, schema), Paimon enriches each dataset with a namespace and a **config facet** containing core table options and Iceberg metadata options.
19+
20+
:::warning
21+
Full Table API lineage support requires <a href="https://github.com/apache/flink/pull/27727" style="color: #1677ff; text-decoration: underline;">this upstream Flink change</a> which is pending merge. Without it, Flink will not call `getLineageVertex()` for `DataStreamScanProvider` and `DataStreamSinkProvider`, which can lead to potentially missing Paimon input or output datasets. This applies to all Flink 2.x versions that do not yet include this change.
22+
:::
23+
24+
## DataStream API
25+
26+
Paimon also supports lineage for the DataStream API. Lineage is automatically reported for sources built via `FlinkSourceBuilder` and for all sinks that extend `FlinkSink`.
27+
28+
Similar to the Table API, it reports all lineage information including name, namespace, schema facet, and config facet.
29+
30+
## Configuration
31+
32+
To consume lineage events, a [JobStatusChangedListener](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/job_status_listener/#configuration) must be added in your Flink configuration.
33+
34+
For example, to configure with OpenLineage:
35+
36+
```
37+
execution.job-status-changed-listeners = io.openlineage.flink.listener.OpenLineageJobStatusChangedListenerFactory
38+
```

docs/sidebars.js

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,8 @@ const sidebars = {
151151
"flink/default-value",
152152
"flink/procedures",
153153
"flink/action-jars",
154-
"flink/savepoint"
154+
"flink/savepoint",
155+
"flink/lineage"
155156
]
156157
},
157158
{

0 commit comments

Comments
 (0)