Skip to content

Commit 07e7ff0

Browse files
Merge pull request #214905 from whhender/adding-manual-lineage
Adding manual lineage preview
2 parents b1e3b8f + 25347b5 commit 07e7ff0

File tree

6 files changed

+36
-3
lines changed

6 files changed

+36
-3
lines changed

articles/purview/catalog-lineage-user-guide.md

Lines changed: 36 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@ This article provides an overview of the data lineage features in Microsoft Purv
1515

1616
One of the platform features of Microsoft Purview is the ability to show the lineage between datasets created by data processes. Systems like Data Factory, Data Share, and Power BI capture the lineage of data as it moves. Custom lineage reporting is also supported via Atlas hooks and REST API.
1717

18-
## Lineage collection
18+
## Lineage collection
19+
1920
Metadata collected in Microsoft Purview from enterprise data systems are stitched across to show an end to end data lineage. Data systems that collect lineage into Microsoft Purview are broadly categorized into following three types:
2021

2122
- [Data processing systems](#data-processing-systems)
@@ -26,7 +27,7 @@ Each system supports a different level of lineage scope. Check the sections belo
2627

2728
### Known limitations
2829

29-
* Database Views used as source of process activity(Azure Data Factory, Synapse Pipelines, Azure SQL Database, Azure Data Share) are currently captured as Database Table objects in Microsoft Purview. If the Database is also scanned, the View assets are discovered separately in Microsoft Purview. In this scenario, two assets with same name captured in Microsoft Purview, one as a Table with data lineage and another as a View.
30+
* Database Views used as source of process activity(Azure Data Factory, Synapse Pipelines, Azure SQL Database, Azure Data Share) are currently captured as Database Table objects in Microsoft Purview. If the Database is also scanned, the View assets are discovered separately in Microsoft Purview. In this scenario, two assets with same name captured in Microsoft Purview, one as a Table with data lineage and another as a View.
3031

3132
### Data processing systems
3233
Data integration and ETL tools can push lineage into Microsoft Purview at execution time. Tools such as Data Factory, Data Share, Synapse, Azure Databricks, and so on, belong to this category of data processing systems. The data processing systems reference datasets as source from different databases and storage solutions to create target datasets. The list of data processing systems currently integrated with Microsoft Purview for lineage are listed in below table.
@@ -90,7 +91,39 @@ To access lineage information for an asset in Microsoft Purview, follow the step
9091

9192
Microsoft Purview supports asset level lineage for the datasets and processes. To see the asset level lineage go to the **Lineage** tab of the current asset in the catalog. Select the current dataset asset node. By default the list of columns belonging to the data appears in the left pane.
9293

93-
:::image type="content" source="./media/catalog-lineage-user-guide/view-columns-from-lineage-inline.png" alt-text="Screenshot showing how to select View columns in the lineage page" lightbox="./media/catalog-lineage-user-guide/view-columns-from-lineage.png"border="true":::
94+
:::image type="content" source="./media/catalog-lineage-user-guide/view-columns-from-lineage-inline.png" alt-text="Screenshot showing how to select View columns in the lineage page." lightbox="./media/catalog-lineage-user-guide/view-columns-from-lineage.png"border="true":::
95+
96+
## Manual lineage (preview)
97+
98+
Data lineage in Microsoft Purview is [automated](#lineage-collection) for many assets in on-premises, multicloud, and SaaS environments. While we continue to add more automated sources, manual lineage allows you to document lineage metadata for sources where automation isn't yet supported, without using any code.
99+
100+
To add manual lineage for any of your assets, follow these steps:
101+
102+
1. [Search for your asset in the data catalog](how-to-search-catalog.md) and select it to view details.
103+
1. Select **Edit** and navigate to the **Lineage** tab.
104+
:::image type="content" source="./media/catalog-lineage-user-guide/select-edit.png" alt-text="Screenshot of a data asset in Microsoft Purview, with the edit option highlighted.":::
105+
:::image type="content" source="./media/catalog-lineage-user-guide/select-lineage.png" alt-text="Screenshot of a data asset edit page, with the Lineage tab highlighted.":::
106+
1. Select **Add Lineage** in the list panel to add an asset as part of the manual lineage.
107+
:::image type="content" source="./media/catalog-lineage-user-guide/add-lineage.png" alt-text="Screenshot of a data asset lineage page, with the add lineage button highlighted.":::
108+
1. Select the relationship type:
109+
1. For upstream lineage - select the relationship as **Consumes**
110+
1. For downstream lineage - select the relationship as **Produces**
111+
1. Select the asset dropdown to find the asset from the suggested list or **View more** to search the full catalog. Select your asset.
112+
:::image type="content" source="./media/catalog-lineage-user-guide/select-asset-dropdown.png" alt-text="Screenshot of a data asset lineage page, with the asset dropdown highlighted.":::
113+
1. Now you can see the lineage relationship. You can add another by selecting the **Add Lineage** button again, or delete the newly added lineage by selecting the trash can icon. When you're finished, select the **Save** button to save your lineage and exit edit mode.
114+
:::image type="content" source="./media/catalog-lineage-user-guide/delete-or-save.png" alt-text="Screenshot of a data asset lineage page, the delete and save buttons highlighted.":::
115+
116+
### Known limitations of manual lineage
117+
118+
* Current asset picker experience allows selecting only one asset at a time.
119+
* Column level manual lineage is currently not supported.
120+
* Data curation access required for both source and target assets.
121+
* These asset types don't currently allow manual lineage because they support automated lineage:
122+
* Azure Data Factory
123+
* Synapse pipelines
124+
* Power BI datasets
125+
* Teradata stored procedure
126+
* Azure SQL stored procedure
94127

95128
## Dataset column lineage
96129

42.7 KB
Loading
60.3 KB
Loading
68.5 KB
Loading
102 KB
Loading
42.9 KB
Loading

0 commit comments

Comments
 (0)