diff --git a/docs/docs/flink/lineage.md b/docs/docs/flink/lineage.md new file mode 100644 index 000000000000..8c4c99008f85 --- /dev/null +++ b/docs/docs/flink/lineage.md @@ -0,0 +1,38 @@ +--- +title: "Data Lineage" +sidebar_position: 13 +--- + +# Data Lineage + +Paimon implements [FLIP-314](https://cwiki.apache.org/confluence/spaces/FLINK/pages/255070913/FLIP-314+Support+Customized+Job+Lineage+Listener) to expose lineage information through Flink's native [LineageVertexProvider](https://nightlies.apache.org/flink/flink-docs-stable/docs/internals/data_lineage/) interface. This allows lineage consumers (such as [OpenLineage](https://openlineage.io/docs/integrations/flink/flink2)) to automatically discover Paimon datasets in the Flink job graph. + +:::info +Data Lineage support is available only for Flink 2.0 and above. +::: + +## Table API / SQL + +When using Paimon tables via Flink SQL or Table API, lineage is automatically reported for both source and sink tables. + +Apart from the table lineage information provided by Flink natively (name, schema), Paimon enriches each dataset with a namespace and a **config facet** containing core table options and Iceberg metadata options. + +:::warning +Full Table API lineage support requires this upstream Flink change which is pending merge. Without it, Flink will not call `getLineageVertex()` for `DataStreamScanProvider` and `DataStreamSinkProvider`, which can lead to potentially missing Paimon input or output datasets. This applies to all Flink 2.x versions that do not yet include this change. +::: + +## DataStream API + +Paimon also supports lineage for the DataStream API. Lineage is automatically reported for sources built via `FlinkSourceBuilder` and for all sinks that extend `FlinkSink`. + +Similar to the Table API, it reports all lineage information including name, namespace, schema facet, and config facet. + +## Configuration + +To consume lineage events, a [JobStatusChangedListener](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/job_status_listener/#configuration) must be added in your Flink configuration. + +For example, to configure with OpenLineage: + +``` +execution.job-status-changed-listeners = io.openlineage.flink.listener.OpenLineageJobStatusChangedListenerFactory +``` diff --git a/docs/sidebars.js b/docs/sidebars.js index 62a0f3ec039c..4223c1347faf 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -151,7 +151,8 @@ const sidebars = { "flink/default-value", "flink/procedures", "flink/action-jars", - "flink/savepoint" + "flink/savepoint", + "flink/lineage" ] }, {