You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/purview/concept-data-lineage.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,15 +13,15 @@ This article provides an overview of data lineage in Microsoft Purview Data Cata
13
13
14
14
- Raw data staged from various platforms
15
15
- Transformed and prepared data
16
-
- Data used by visualization platforms.
16
+
- Data used by visualization platforms
17
17
18
18
## Use cases
19
19
20
20
Data lineage is broadly understood as the lifecycle that spans the data’s origin, and where it moves over time across the data estate. It's used for different kinds of backwards-looking scenarios such as troubleshooting, tracing root cause in data pipelines and debugging. Lineage is also used for data quality analysis, compliance and “what if” scenarios often referred to as impact analysis. Lineage is represented visually to show data moving from source to destination including how the data was transformed. Given the complexity of most enterprise data environments, these views can be hard to understand without doing some consolidation or masking of peripheral data points.
21
21
22
22
## Lineage experience in Microsoft Purview Data Catalog
23
23
24
-
Microsoft Purview Data Catalog will connect with other data processing, storage, and analytics systems to extract lineage information. The information is combined to represent a generic, scenario-specific lineage experience in the Catalog.
24
+
Microsoft Purview Data Catalog will connect with other data processing, storage, and analytics systems to extract lineage information. The information is combined to represent a generic, scenario-specific lineage experience in the catalog.
25
25
26
26
:::image type="content" source="media/concept-lineage/lineage-end-end-inline.png" alt-text="end-end lineage showing data copied from blob store all the way to Power BI dashboard" lightbox="media/concept-lineage/lineage-end-end.png":::
Copy file name to clipboardExpand all lines: articles/purview/concept-scans-and-ingestion.md
+5-3Lines changed: 5 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,8 +16,8 @@ This article provides an overview of the Scanning and Ingestion features in Micr
16
16
17
17
-[**Scanning**](#scanning) captures metadata from [data sources](microsoft-purview-connector-overview.md) and brings it to Microsoft Purview.
18
18
-[**Ingestion**](#ingestion) processes metadata and stores it in the data catalog from both:
19
-
- Data source scans
20
-
- Lineage connections
19
+
- Data source scans - scanned metadata is added to the Microsoft Purview Data Map.
20
+
- Lineage connections - transformation resources add metadata about their sources, outputs, and activities to the Microsoft Purview Data Map.
21
21
22
22
## Scanning
23
23
@@ -80,7 +80,9 @@ The technical metadata or classifications identified by the scanning process are
80
80
81
81
## Ingestion from lineage connections
82
82
83
-
Resources like [Azure Data Factory](how-to-link-azure-data-factory.md) and [Azure Synapse](how-to-lineage-azure-synapse-analytics.md) can be connected to Microsoft Purview to bring lineage information into your Microsoft Purview Data Map. For example, when a copy pipeline runs in an Azure Data Factory that has been connected to Microsoft Purview, metadata about inputs, the activity, and outputs are ingested in Microsoft Purview and the information is added to the data map.
83
+
Resources like [Azure Data Factory](how-to-link-azure-data-factory.md) and [Azure Synapse](how-to-lineage-azure-synapse-analytics.md) can be connected to Microsoft Purview to bring data source and lineage information into your Microsoft Purview Data Map. For example, when a copy pipeline runs in an Azure Data Factory that has been connected to Microsoft Purview, metadata about the input sources, the activity, and the output sources are ingested in Microsoft Purview and the information is added to the data map.
84
+
85
+
If a data source has already been added to the data map through a scan, lineage information about the activity will be added to the existing source. If the data source hasn't yet been added to the data map, the lineage ingestion process will add it to the root collection with its lineage information.
84
86
85
87
For more information about the available lineage connections, see the [lineage user guide](catalog-lineage-user-guide.md).
Copy file name to clipboardExpand all lines: articles/purview/how-to-lineage-azure-synapse-analytics.md
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,11 +6,17 @@ ms.author: jingwang
6
6
ms.service: purview
7
7
ms.subservice: purview-data-catalog
8
8
ms.topic: how-to
9
-
ms.date: 12/14/2022
9
+
ms.date: 03/13/2023
10
10
---
11
11
# How to get lineage from Azure Synapse Analytics into Microsoft Purview
12
12
13
-
This document explains the steps required for connecting an Azure Synapse workspace with a Microsoft Purview account to track data lineage. The document also gets into the details of the coverage scope and supported lineage capabilities.
13
+
This document explains the steps required for connecting an Azure Synapse workspace with a Microsoft Purview account to track [data lineage](concept-data-lineage.md) and [ingest data sources](concept-scans-and-ingestion.md#ingestion). The document also gets into the details of the activity coverage scope and supported lineage capabilities.
14
+
15
+
When you connect Azure Synapse Analytics to Microsoft Purview, whenever a [supported pipeline activity](#supported-azure-synapse-capabilities) is run, metadata about the activity's source data, output data, and the activity will be automatically [ingested](concept-scans-and-ingestion.md#ingestion) into the Microsoft Purview Data Map.
16
+
17
+
If a data source has already been scanned and exists in the data map, the ingestion process will add the lineage information from Azure Synapse Analytics to that existing source. If the source or output doesn't exist in the data map and is [supported by Azure Synapse Analytics lineage](#supported-azure-synapse-capabilities) Microsoft Purview will automatically add their metadata from Synapse Analytics into the data map under the root collection.
18
+
19
+
This can be an excellent way to monitor your data estate as users move and transform information using Azure Synapse Analytics.
14
20
15
21
## Supported Azure Synapse capabilities
16
22
@@ -22,10 +28,10 @@ Currently, Microsoft Purview captures runtime lineage from the following Azure S
22
28
> [!IMPORTANT]
23
29
> Microsoft Purview drops lineage if the source or destination uses an unsupported data storage system.
If your Microsoft Purview account is protected by firewall, learn how to let Azure Synapse [access a secured Microsoft Purview account](../synapse-analytics/catalog-and-governance/how-to-access-secured-purview-account.md) through Microsoft Purview private endpoints.
30
36
31
37
## Bring Azure Synapse lineage into Microsoft Purview
Copy file name to clipboardExpand all lines: articles/purview/how-to-link-azure-data-factory.md
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,11 +6,17 @@ ms.author: jingwang
6
6
ms.service: purview
7
7
ms.subservice: purview-data-catalog
8
8
ms.topic: how-to
9
-
ms.date: 12/14/2022
9
+
ms.date: 03/13/2023
10
10
---
11
11
# How to connect Azure Data Factory and Microsoft Purview
12
12
13
-
This document explains the steps required for connecting an Azure Data Factory account with a Microsoft Purview account to track data lineage. The document also gets into the details of the coverage scope and supported lineage patterns.
13
+
This document explains the steps required for connecting an Azure Data Factory account with a Microsoft Purview account to track [data lineage](concept-data-lineage.md) and [ingest data sources](concept-scans-and-ingestion.md#ingestion). The document also gets into the details of the activity coverage scope and supported lineage patterns.
14
+
15
+
When you connect an Azure Data Factory to Microsoft Purview, whenever a [supported Azure Data Factory activity](#supported-azure-data-factory-activities) is run, metadata about the activity's source data, output data, and the activity will be automatically [ingested](concept-scans-and-ingestion.md#ingestion) into the Microsoft Purview Data Map.
16
+
17
+
If a data source has already been scanned and exists in the data map, the ingestion process will add the lineage information from Azure Data Factory to that existing source. If the source or output doesn't exist in the data map and is [supported by Azure Data Factory lineage](#supported-azure-data-factory-activities) Microsoft Purview will automatically add their metadata from Azure Data Factory into the data map under the root collection.
18
+
19
+
This can be an excellent way to monitor your data estate as users move and transform information using Azure Data Factory.
14
20
15
21
## View existing Data Factory connections
16
22
@@ -91,14 +97,14 @@ Microsoft Purview captures runtime lineage from the following Azure Data Factory
91
97
92
98
The integration between Data Factory and Microsoft Purview supports only a subset of the data systems that Data Factory supports, as described in the following sections.
Refer to [supported data stores](how-to-lineage-sql-server-integration-services.md#supported-data-stores).
99
105
100
106
## Access secured Microsoft Purview account
101
-
107
+
102
108
If your Microsoft Purview account is protected by firewall, learn how to let Data Factory [access a secured Microsoft Purview account](../data-factory/how-to-access-secured-purview-account.md) through Microsoft Purview private endpoints.
103
109
104
110
## Bring Data Factory lineage into Microsoft Purview
0 commit comments