Skip to content

Commit 0ba2262

Browse files
committed
Ingestion updates
1 parent 227128f commit 0ba2262

File tree

4 files changed

+27
-13
lines changed

4 files changed

+27
-13
lines changed

articles/purview/concept-data-lineage.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,15 @@ This article provides an overview of data lineage in Microsoft Purview Data Cata
1313

1414
- Raw data staged from various platforms
1515
- Transformed and prepared data
16-
- Data used by visualization platforms.
16+
- Data used by visualization platforms
1717

1818
## Use cases
1919

2020
Data lineage is broadly understood as the lifecycle that spans the data’s origin, and where it moves over time across the data estate. It's used for different kinds of backwards-looking scenarios such as troubleshooting, tracing root cause in data pipelines and debugging. Lineage is also used for data quality analysis, compliance and “what if” scenarios often referred to as impact analysis. Lineage is represented visually to show data moving from source to destination including how the data was transformed. Given the complexity of most enterprise data environments, these views can be hard to understand without doing some consolidation or masking of peripheral data points.
2121

2222
## Lineage experience in Microsoft Purview Data Catalog
2323

24-
Microsoft Purview Data Catalog will connect with other data processing, storage, and analytics systems to extract lineage information. The information is combined to represent a generic, scenario-specific lineage experience in the Catalog.
24+
Microsoft Purview Data Catalog will connect with other data processing, storage, and analytics systems to extract lineage information. The information is combined to represent a generic, scenario-specific lineage experience in the catalog.
2525

2626
:::image type="content" source="media/concept-lineage/lineage-end-end-inline.png" alt-text="end-end lineage showing data copied from blob store all the way to Power BI dashboard" lightbox="media/concept-lineage/lineage-end-end.png":::
2727

articles/purview/concept-scans-and-ingestion.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ This article provides an overview of the Scanning and Ingestion features in Micr
1616

1717
- [**Scanning**](#scanning) captures metadata from [data sources](microsoft-purview-connector-overview.md) and brings it to Microsoft Purview.
1818
- [**Ingestion**](#ingestion) processes metadata and stores it in the data catalog from both:
19-
- Data source scans
20-
- Lineage connections
19+
- Data source scans - scanned metadata is added to the Microsoft Purview Data Map.
20+
- Lineage connections - transformation resources add metadata about their sources, outputs, and activities to the Microsoft Purview Data Map.
2121

2222
## Scanning
2323

@@ -80,7 +80,9 @@ The technical metadata or classifications identified by the scanning process are
8080

8181
## Ingestion from lineage connections
8282

83-
Resources like [Azure Data Factory](how-to-link-azure-data-factory.md) and [Azure Synapse](how-to-lineage-azure-synapse-analytics.md) can be connected to Microsoft Purview to bring lineage information into your Microsoft Purview Data Map. For example, when a copy pipeline runs in an Azure Data Factory that has been connected to Microsoft Purview, metadata about inputs, the activity, and outputs are ingested in Microsoft Purview and the information is added to the data map.
83+
Resources like [Azure Data Factory](how-to-link-azure-data-factory.md) and [Azure Synapse](how-to-lineage-azure-synapse-analytics.md) can be connected to Microsoft Purview to bring data source and lineage information into your Microsoft Purview Data Map. For example, when a copy pipeline runs in an Azure Data Factory that has been connected to Microsoft Purview, metadata about the input sources, the activity, and the output sources are ingested in Microsoft Purview and the information is added to the data map.
84+
85+
If a data source has already been added to the data map through a scan, lineage information about the activity will be added to the existing source. If the data source hasn't yet been added to the data map, the lineage ingestion process will add it to the root collection with its lineage information.
8486

8587
For more information about the available lineage connections, see the [lineage user guide](catalog-lineage-user-guide.md).
8688

articles/purview/how-to-lineage-azure-synapse-analytics.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,17 @@ ms.author: jingwang
66
ms.service: purview
77
ms.subservice: purview-data-catalog
88
ms.topic: how-to
9-
ms.date: 12/14/2022
9+
ms.date: 03/13/2023
1010
---
1111
# How to get lineage from Azure Synapse Analytics into Microsoft Purview
1212

13-
This document explains the steps required for connecting an Azure Synapse workspace with a Microsoft Purview account to track data lineage. The document also gets into the details of the coverage scope and supported lineage capabilities.
13+
This document explains the steps required for connecting an Azure Synapse workspace with a Microsoft Purview account to track [data lineage](concept-data-lineage.md) and [ingest data sources](concept-scans-and-ingestion.md#ingestion). The document also gets into the details of the activity coverage scope and supported lineage capabilities.
14+
15+
When you connect Azure Synapse Analytics to Microsoft Purview, whenever a [supported pipeline activity](#supported-azure-synapse-capabilities) is run, metadata about the activity's source data, output data, and the activity will be automatically [ingested](concept-scans-and-ingestion.md#ingestion) into the Microsoft Purview Data Map.
16+
17+
If a data source has already been scanned and exists in the data map, the ingestion process will add the lineage information from Azure Synapse Analytics to that existing source. If the source or output doesn't exist in the data map and is [supported by Azure Synapse Analytics lineage](#supported-azure-synapse-capabilities) Microsoft Purview will automatically add their metadata from Synapse Analytics into the data map under the root collection.
18+
19+
This can be an excellent way to monitor your data estate as users move and transform information using Azure Synapse Analytics.
1420

1521
## Supported Azure Synapse capabilities
1622

@@ -22,10 +28,10 @@ Currently, Microsoft Purview captures runtime lineage from the following Azure S
2228
> [!IMPORTANT]
2329
> Microsoft Purview drops lineage if the source or destination uses an unsupported data storage system.
2430
25-
[!INCLUDE[azure-synapse-supported-activity-lineage-capabilities](includes/data-factory-common-supported-capabilities.md)]
31+
[!INCLUDE[azure-synapse-supported-activity-lineage-capabilities](includes/data-factory-common-supported-capabilities.md)
2632

2733
## Access secured Microsoft Purview account
28-
34+
2935
If your Microsoft Purview account is protected by firewall, learn how to let Azure Synapse [access a secured Microsoft Purview account](../synapse-analytics/catalog-and-governance/how-to-access-secured-purview-account.md) through Microsoft Purview private endpoints.
3036

3137
## Bring Azure Synapse lineage into Microsoft Purview

articles/purview/how-to-link-azure-data-factory.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,17 @@ ms.author: jingwang
66
ms.service: purview
77
ms.subservice: purview-data-catalog
88
ms.topic: how-to
9-
ms.date: 12/14/2022
9+
ms.date: 03/13/2023
1010
---
1111
# How to connect Azure Data Factory and Microsoft Purview
1212

13-
This document explains the steps required for connecting an Azure Data Factory account with a Microsoft Purview account to track data lineage. The document also gets into the details of the coverage scope and supported lineage patterns.
13+
This document explains the steps required for connecting an Azure Data Factory account with a Microsoft Purview account to track [data lineage](concept-data-lineage.md) and [ingest data sources](concept-scans-and-ingestion.md#ingestion). The document also gets into the details of the activity coverage scope and supported lineage patterns.
14+
15+
When you connect an Azure Data Factory to Microsoft Purview, whenever a [supported Azure Data Factory activity](#supported-azure-data-factory-activities) is run, metadata about the activity's source data, output data, and the activity will be automatically [ingested](concept-scans-and-ingestion.md#ingestion) into the Microsoft Purview Data Map.
16+
17+
If a data source has already been scanned and exists in the data map, the ingestion process will add the lineage information from Azure Data Factory to that existing source. If the source or output doesn't exist in the data map and is [supported by Azure Data Factory lineage](#supported-azure-data-factory-activities) Microsoft Purview will automatically add their metadata from Azure Data Factory into the data map under the root collection.
18+
19+
This can be an excellent way to monitor your data estate as users move and transform information using Azure Data Factory.
1420

1521
## View existing Data Factory connections
1622

@@ -91,14 +97,14 @@ Microsoft Purview captures runtime lineage from the following Azure Data Factory
9197
9298
The integration between Data Factory and Microsoft Purview supports only a subset of the data systems that Data Factory supports, as described in the following sections.
9399

94-
[!INCLUDE[data-factory-supported-lineage-capabilities](includes/data-factory-common-supported-capabilities.md)]
100+
[!INCLUDE[data-factory-supported-lineage-capabilities](includes/data-factory-common-supported-capabilities.md)
95101

96102
### Execute SSIS Package support
97103

98104
Refer to [supported data stores](how-to-lineage-sql-server-integration-services.md#supported-data-stores).
99105

100106
## Access secured Microsoft Purview account
101-
107+
102108
If your Microsoft Purview account is protected by firewall, learn how to let Data Factory [access a secured Microsoft Purview account](../data-factory/how-to-access-secured-purview-account.md) through Microsoft Purview private endpoints.
103109

104110
## Bring Data Factory lineage into Microsoft Purview

0 commit comments

Comments
 (0)