Merge pull request #221618 from linda33wj/main

Add link monitoring doc

Author: BCS2022

articles/purview/.openpublishing.redirection.purview.json

@@ -239,6 +239,11 @@
         "source_path_from_root": "/articles/purview/concept-default-purview-account.md",
         "redirect_url": "/azure/purview/concept-best-practices-accounts",
         "redirect_document_id": true
+      },
+      {
+        "source_path_from_root": "/articles/purview/how-to-monitor-scan-runs.md",
+        "redirect_url": "/azure/purview/how-to-monitor-data-map-population",
+        "redirect_document_id": true
       }
 ]
 }

articles/purview/how-to-lineage-azure-synapse-analytics.md

@@ -6,7 +6,7 @@ ms.author: jingwang
 ms.service: purview
 ms.subservice: purview-data-catalog
 ms.topic: how-to
-ms.date: 12/05/2022
+ms.date: 12/14/2022
 ---
 # How to get lineage from Azure Synapse Analytics into Microsoft Purview
 
@@ -54,6 +54,10 @@ Select the Synapse account -> pipeline -> activity, you can view the lineage inf
 
 :::image type="content" source="./media/how-to-lineage-azure-synapse-analytics/browse-azure-synapse-pipeline-lineage.png" alt-text="Browse the Azure Synapse pipeline lineage in Microsoft Purview." lightbox="./media/how-to-lineage-azure-synapse-analytics/browse-azure-synapse-pipeline-lineage.png":::
 
+## Monitor the Azure Synapse Analytics links
+
+In Microsoft Purview governance portal, you can [monitor the Azure Synapse Analytics links](how-to-monitor-data-map-population.md#monitor-links).
+
 ## Next steps
 
 [Catalog lineage user guide](catalog-lineage-user-guide.md)

articles/purview/how-to-link-azure-data-factory.md

@@ -6,7 +6,7 @@ ms.author: jingwang
 ms.service: purview
 ms.subservice: purview-data-catalog
 ms.topic: how-to
-ms.date: 12/05/2022
+ms.date: 12/14/2022
 ---
 # How to connect Azure Data Factory and Microsoft Purview
 
@@ -65,7 +65,7 @@ Data factory's managed identity is used to authenticate lineage push operations
 
 Grant the data factory's managed identity **Data Curator** role on Microsoft Purview **root collection**. Learn more about [Access control in Microsoft Purview](../purview/catalog-permissions.md) and [Add roles and restrict access through collections](../purview/how-to-create-and-manage-collections.md#add-roles-and-restrict-access-through-collections).
 
-### Remove data factory connections
+### Remove Data Factory connections
 
 To remove a data factory connection, do the following:
 
@@ -74,6 +74,10 @@ To remove a data factory connection, do the following:
 
     :::image type="content" source="./media/how-to-link-azure-data-factory/remove-data-factory-connection.png" alt-text="Screenshot showing how to select data factories to remove connection." lightbox="./media/how-to-link-azure-data-factory/remove-data-factory-connection.png":::
 
+## Monitor the Data Factory links
+
+In Microsoft Purview governance portal, you can [monitor the Data Factory links](how-to-monitor-data-map-population.md#monitor-links).
+
 ## Supported Azure Data Factory activities
 
 Microsoft Purview captures runtime lineage from the following Azure Data Factory activities:

articles/purview/how-to-monitor-scan-runs.md renamed to articles/purview/how-to-monitor-data-map-population.md

@@ -1,17 +1,17 @@
 ---
-title: Monitor scan runs in Microsoft Purview
-description: This guide describes how to monitor the scan runs in Microsoft Purview. 
+title: Monitor data map population in Microsoft Purview
+description: This guide describes how to monitor data map population including the scan runs and links in Microsoft Purview. 
 author: linda33wj
 ms.author: jingwang
 ms.service: purview
 ms.subservice: purview-data-map
 ms.topic: how-to
-ms.date: 12/08/2022
+ms.date: 12/14/2022
 ---
 
-# Monitor scan runs in Microsoft Purview
+# Monitor data map population in Microsoft Purview
 
-In Microsoft Purview, you can register and scan various types of data sources, and you can view the scan status over time. This article outlines how to monitor and get a bird's eye view of your scan runs in Microsoft Purview.
+In Microsoft Purview, you can scan various types of data sources and view the scan status over time; you can also connect other services with Microsoft Purview and view the trends of the ingested assets/relationship. This article outlines how to monitor and get a bird's eye view of the data map population.
 
 ## Monitor scan runs
 
@@ -67,7 +67,7 @@ You can click the **run ID** to check more about the scan run details:
     - **Assets classified**: The number of assets sampled to classify the data, regardless of whether the assets have any matching classification or not. It's a subset of the discovered assets based on the [sampling mechanism](microsoft-purview-connector-overview.md#sampling-data-for-classification). For incremental scan, only newly created or updated assets may be selected for classification.
     - **Duration**: The scan phase duration and the start/end time.
 
-- **Data ingestion** section summarizes the metrics for ingestion phase that Purview populates the data map with the identified metadata and lineage.
+- **Data ingestion** section summarizes the metrics for ingestion phase that Purview populates the data map with the identified metadata and relationship.
 
     - **Ingestion status**:
 
@@ -80,12 +80,43 @@ You can click the **run ID** to check more about the scan run details:
         | In Progress         | The ingestion is running in progress.                        |
         | Queued              | The ingestion is waiting for available service resource or waiting for scan to discover metadata. |
 
-    - **Assets ingested**: The number of assets ingested into the data map. For full scan, the number is equal to the "assets discovered" count; for incremental scan, it only includes the newly created or updated assets, in which case may be less than the "assets discovered" count. When scanning file-based source, it's the raw assets count before resource set aggregation.
+    - **Assets ingested**: The number of assets ingested into the data map. For incremental scan, it only includes the newly created or updated assets, in which case may be less than the "assets discovered" count. When scanning file-based source, it's the raw assets count before resource set aggregation.
 
     - **Relationships ingested**: The number of relationships ingested into the data map. It includes lineage and other relationships like foreign key relationships.
 
     - **Duration**: The ingestion duration and the start/end time.
 
+## Monitor links
+
+You can connect other services with Microsoft Purview to establish a "link", which will make the metadata and lineage of that service's assets available to Microsoft Purview. Currently, link is supported for [Azure Data Factory](how-to-link-azure-data-factory.md) and [Azure Synapse Analytics](how-to-lineage-azure-synapse-analytics.md).
+
+To monitor the assets and relationship ingested over the links:
+
+1. Go to your Microsoft Purview account -> open **Microsoft Purview governance portal** -> **Data map** -> **Monitoring** -> **Links**. You need to have **Data source admin** role on any collection to access the Monitoring tab. And you'll see the results that belong to the collections on which you have data source admin privilege. Permission on root collection is needed to monitor Azure Data Factory and Azure Synapse Analytics links.
+
+1. You can see the high-level KPIs including total number of sources, number of ingested assets and relationship (lineage), followed by trending charts over time. You can apply additional filters on the following to narrow down the results:
+
+    - Source type
+    - Source name
+    - Date range: Default is 30 days. You can also choose last seven days or a custom date range. The retention is 45 days.
+
+    The metrics are reported till the date time shown at the top right corner. And the aggregation will happen hourly.
+
+    :::image type="content" source="./media/how-to-monitor-scan-runs/monitor-links.png" alt-text="Screenshot of view link results."  lightbox="./media/how-to-monitor-scan-runs/monitor-links.png":::
+
+1. At the bottom of the graph, there's a **View more** link for you to explore further. In the **Link status** page, you can see a list of source names along with the source type, assets ingested, relationship ingested and the last run date time. The filters in the previous page will be carried over, and you can further filter the list by source type, source name and date range.
+
+    :::image type="content" source="./media/how-to-monitor-scan-runs/monitor-links-drilldown.png" alt-text="Screenshot of view link results by source."  lightbox="./media/how-to-monitor-scan-runs/monitor-links-drilldown.png":::
+
+1. You can drill down to each source to see the next level details by clicking the source name. For example, for Azure Data Factory, it shows how each pipeline activity reports the assets and relationship to Microsoft Purview, with the name in the format of `<pipeline_name>/<activity_name>`.
+
+    :::image type="content" source="./media/how-to-monitor-scan-runs/monitor-links-drilldown-second-layer.png" alt-text="Screenshot of view link results by source's sub-artifacts."  lightbox="./media/how-to-monitor-scan-runs/monitor-links-drilldown-second-layer.png":::
+
+### Known limitations
+
+- For Azure Data Factory and Azure Synapse Analytics, currently this link monitoring captures the assets and relationship generated from copy activity, but not data flow and SSIS activities.
+- The aggregation and date filter are in UTC time.
+
 ## Scans no longer run
 
 If your Microsoft Purview scan used to successfully run, but are now failing, check these things:

articles/purview/media/how-to-monitor-scan-runs/monitor-links-drilldown-second-layer.png

@@ -247,8 +247,8 @@ items:
       href: create-a-scan-rule-set.md
     - name: Resource set pattern rules
       href: how-to-resource-set-pattern-rules.md
-    - name: Monitor scan runs
-      href: how-to-monitor-scan-runs.md
+    - name: Monitor data map population
+      href: how-to-monitor-data-map-population.md
     - name: Troubleshoot connection issues
       href: troubleshoot-connections.md
   - name: Data catalog