Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/docs/flink/lineage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "Data Lineage"
sidebar_position: 13
---

# Data Lineage

Paimon implements [FLIP-314](https://cwiki.apache.org/confluence/spaces/FLINK/pages/255070913/FLIP-314+Support+Customized+Job+Lineage+Listener) to expose lineage information through Flink's native [LineageVertexProvider](https://nightlies.apache.org/flink/flink-docs-stable/docs/internals/data_lineage/) interface. This allows lineage consumers (such as [OpenLineage](https://openlineage.io/docs/integrations/flink/flink2)) to automatically discover Paimon datasets in the Flink job graph.

:::info
Data Lineage support is available only for Flink 2.0 and above.
:::

## Table API / SQL

When using Paimon tables via Flink SQL or Table API, lineage is automatically reported for both source and sink tables.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads as unconditional, but the note below says the Flink planner change needed to extract Paimon datasets from DataStreamScanProvider/DataStreamSinkProvider is still pending. Could we scope this to Flink versions that contain that upstream change, or phrase Table API/SQL lineage as partial until that PR is merged and released? Otherwise users may configure the listener and still get events missing the Paimon datasets.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JingsongLi I have updated the note section and made it to a warning so that it is more explicit, can you please let me know if this is okay?

It will show as:

Image


Apart from the table lineage information provided by Flink natively (name, schema), Paimon enriches each dataset with a namespace and a **config facet** containing core table options and Iceberg metadata options.

:::warning
Full Table API lineage support requires <a href="https://github.com/apache/flink/pull/27727" style="color: #1677ff; text-decoration: underline;">this upstream Flink change</a> which is pending merge. Without it, Flink will not call `getLineageVertex()` for `DataStreamScanProvider` and `DataStreamSinkProvider`, which can lead to potentially missing Paimon input or output datasets. This applies to all Flink 2.x versions that do not yet include this change.
:::

## DataStream API

Paimon also supports lineage for the DataStream API. Lineage is automatically reported for sources built via `FlinkSourceBuilder` and for all sinks that extend `FlinkSink`.

Similar to the Table API, it reports all lineage information including name, namespace, schema facet, and config facet.

## Configuration

To consume lineage events, a [JobStatusChangedListener](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/job_status_listener/#configuration) must be added in your Flink configuration.

For example, to configure with OpenLineage:

```
execution.job-status-changed-listeners = io.openlineage.flink.listener.OpenLineageJobStatusChangedListenerFactory
```
3 changes: 2 additions & 1 deletion docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ const sidebars = {
"flink/default-value",
"flink/procedures",
"flink/action-jars",
"flink/savepoint"
"flink/savepoint",
"flink/lineage"
]
},
{
Expand Down
Loading