Merge pull request #223045 from n0elleli/cdcupdate

Cdcupdate

Author: v-regandowner

articles/data-factory/TOC.yml

@@ -296,8 +296,12 @@ items:
       - name: Supported functions
         href: wrangling-functions.md
         displayName: power
-  - name: Change Data Capture
-    href: concepts-change-data-capture.md
+  - name: Change data capture
+    items:
+    - name: Change data capture
+      href: concepts-change-data-capture.md    
+    - name: Change data capture resource
+      href: concepts-change-data-capture-resource.md
   - name: Roles and permissions
     href: concepts-roles-permissions.md
   - name: Naming rules
@@ -354,6 +358,8 @@ items:
       href: author-management-hub.md
     - name: Source control
       href: source-control.md
+    - name: Author a change data capture resource
+      href: how-to-change-data-capture-resource.md
     - name: Connect to Azure DevOps in another tenant
       href: cross-tenant-connections-to-azure-devops.md
     - name: Continuous integration and delivery
@@ -1169,6 +1175,11 @@ items:
     - name: Activities
       href: data-factory-troubleshoot-guide.md
       displayName: timeout, troubleshooting
+    - name: Change data capture
+      items:
+      - name: Change data capture troubleshooting
+        href: change-data-capture-troubleshoot.md
+        displayName: change data capture, troubleshooting
     - name: Connectors
       items:
       - name: Overview and general copy activity errors

articles/data-factory/change-data-capture-troubleshoot.md

@@ -0,0 +1,83 @@
+---
+title: Troubleshoot the change data capture resource
+titleSuffix: Azure Data Factory
+description: Learn how to troubleshoot issues with the change data capture resoure in Azure Data Factory. 
+author: n0elleli
+ms.service: data-factory
+ms.subservice:
+ms.topic: troubleshooting
+ms.date: 01/19/2023
+ms.author: noelleli
+ms.custom:
+---
+
+# Troubleshoot the Change data capture resource in Azure Data Factory
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+This article provides suggestions on how to troubleshoot common problems with the change data capture resource in Azure Data Factory.
+
+## Issue: Trouble enabling native CDC in my SQL source. 
+
+For sources in SQL, two sets of tables are available: tables with native SQL CDC enabled and tables with time-based incremental columns. 
+
+Follow these steps to configure native CDC for a specific source table in your SQL database.
+
+Consider you have following table, with ID as the Primary Key. If a Primary Key is present in the schema, supports_net_changes is set to true by default. If not, configure it using the script in Query 3.  
+
+**Query 1**
+```sql
+
+CREATE TABLE Persons (
+	ID int,
+	LastName varchar(255) NOT NULL,
+	FirstName varchar(255),
+	Age int,
+	Last_login DATETIME,
+    	PRIMARY KEY (ID));
+
+```
+
+ > [!NOTE]
+ > Currently the ADF CDC resource only loads net changes for insert, update and delete operations.
+
+To enable CDC at the database level, execute the following query: 
+
+**Query 2**
+
+```sql
+EXEC sys.sp_cdc_enable_db
+```
+To enable CDC at the table level, execute the following query: 
+
+**Query 3**
+
+```sql
+EXEC sys.sp_cdc_enable_table  
+	@source_schema = N'dbo'  
+	, @source_name = N'Persons'  
+	, @role_name = N'cdc_admin'  
+	, @supports_net_changes = 1
+        , @captured_column_list = N'ID';
+```
+    
+## Issue: Tables are unavailable to select in the CDC resource configuration process. 
+  
+If your SQL source doesn't have SQL Server CDC with net_changed enabled or doesn't have any time-based incremental columns, then the tables in your source will be unavailable for selection. 
+
+## Issue: The debug cluster is not available from a warm pool.
+
+The debug cluster is not available from a warm pool. There will be a wait time in the order of 1+ minutes. 
+
+## Issue: My CDC resource has both source and target linked services that use custom integration runtimes and it won't work. 
+
+In factories with virtual networks, CDC resources will work fine if either the source or target linked service is tied to an auto-resolve integration runtime. If both the source and target linked services use custom integration runtimes, the CDC resource will not work. 
+
+In non-virtual network factories, CDC resources requiring a virtual network will not work. This fix is in progress. 
+
+## Issue: Creating a new linked service pointing to an Azure Key Vault linked service causes an error. 
+
+If you create a new linked service using the CDC fly-out process that points to an Azure Key Vault linked service, the CDC resource will break. This fix is in progress. 
+
+## Next steps
+- [Learn more about the change data capture resource](concepts-change-data-capture-resource.md)
+- [Set up a change data capture resource](how-to-change-data-capture-resource.md)

articles/data-factory/concepts-change-data-capture-resource.md

@@ -0,0 +1,60 @@
+---
+title: Change Data Capture Resource
+titleSuffix: Azure Data Factory
+description: Learn more about the change data capture resource in Azure Data Factory.
+author: n0elleli
+ms.author: noelleli
+ms.reviewer:
+ms.service: data-factory
+ms.subservice: data-movement
+ms.custom:
+ms.topic: conceptual
+ms.date: 01/20/2023
+---
+
+# Change data capture resource overview
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+Adapting to the cloud-first big data world can be incredibly challenging for data engineers who are responsible for building complex data integration and ETL pipelines. 
+
+Azure Data Factory is introducing a new mechanism to make the life of a data engineer easier. 
+
+By automatically detecting data changes at the source without requiring complex designing or coding, ADF is making it a breeze to scale these processes. Change Data Capture will now exist as a **new native top-level resource** in the Azure Data Factory studio where data engineers can quickly configure continuously running jobs to process big data at scale with extreme efficiency. 
+
+The new Change Data Capture resource in ADF allows for full fidelity change data capture that continuously runs in near real-time through a guided configuration experience. 
+
+:::image type="content" source="media/adf-cdc/change-data-capture-resource-1.png" alt-text="Screenshot of new top-level resource in Factory Resources panel.":::
+
+## Supported data sources
+
+* Avro
+* Azure Cosmos DB (SQL API)
+* Azure SQL Database
+* Delimited Text
+* JSON
+* ORC
+* Parquet
+* SQL Server
+* XML
+
+## Supported targets
+
+* Avro
+* Azure SQL Database
+* Azure Synapse Analytics
+* Delimited Text
+* Delta
+* JSON
+* ORC
+* Parquet
+
+## Known limitations
+* Currently, when creating source/target mappings, each source and target is only allowed to be used once. 
+* Continuous, real-time streaming is coming soon.
+* Allow schema drift is coming soon.
+
+For more information on known limitations and troubleshooting assistance, please reference [this troubleshooting guide](change-data-capture-troubleshoot.md).
+
+
+## Next steps
+- [Learn how to set up a change data capture resource](how-to-change-data-capture-resource.md).

articles/data-factory/concepts-change-data-capture.md

@@ -52,6 +52,13 @@ The newly updated rows or updated files can be automatically detected and extrac
 -   [Azure Database for PostgreSQL](connector-azure-database-for-postgresql.md)
 -   [Common data model](format-common-data-model.md)
 
+### Change data capture top-level resource
+
+A new top-level change data capture resource guides users through a simple configuration process to create a resource that will continuously and automatically read changes from data source(s) without needing to design pipelines, data flows, or set up triggers. 
+
+:::image type="content" source="media/adf-cdc/change-data-capture-resource-1.png" alt-text="Screenshot of new top-level artifact in Factory Resources panel.":::
+
+
 ### Customer managed delta data extraction in pipeline
 
 You can always build your own delta data extraction pipeline for all ADF supported data stores including using lookup activity to get the watermark value stored in an external control table, copy activity or mapping data flow activity to query the delta data against timestamp or ID column, and SP activity to write the new watermark value back to your external control table for the next run.  When you want to load new files only from a storage store, you can either delete files every time after they have been moved to the destination successfully, or leverage the time partitioned folder or file names or last modified time to identify the new files. 
@@ -96,3 +103,4 @@ The followings are the templates to use the change data capture in Azure Data Fa
 ## Next steps
 
 - [Learn how to use the checkpoint key in the data flow activity](control-flow-execute-data-flow-activity.md).
+- [Learn about the Change data capture resource in ADF](concepts-change-data-capture-resource.md).

articles/data-factory/how-to-change-data-capture-resource.md

@@ -0,0 +1,173 @@
+---
+title: Capture changed data with a change data capture resource
+description: This tutorial provides step-by-step instructions on how to capture changed data from ADLS Gen2 to SQL DB using a Change data capture resource.
+author: n0elleli
+ms.author: noelleli
+ms.reviewer: 
+ms.service: data-factory
+ms.subservice:
+ms.topic: conceptual
+ms.custom: seo-lt-2019
+ms.date: 01/20/2023
+---
+
+# How to capture changed data from ADLS Gen2 to SQL DB using a Change data capture resource
+[!INCLUDE[appliesto-adf-asa-md]]
+
+In this tutorial, you will use the Azure Data Factory user interface (UI) to create a new Change data capture resource that picks up changed data from an Azure Data Lake Storage (ADLS) Gen2 source to a SQL Database. The configuration pattern in this tutorial can be modified and expanded upon. 
+
+In this tutorial, you follow these steps:
+* Create a change data capture resource.
+* Monitor change data capture activity.
+
+## Pre-requisites
+
+* **Azure subscription.** If you don't have an Azure subscription, create a free Azure account before you begin.
+* **Azure storage account.** You use ADLS storage as a source data store. If you don't have a storage account, see Create an Azure storage account for steps to create one.
+* **Azure SQL Database.** You will use Azure SQL DB as a target data store. If you don’t have a SQL DB, please create one in the Azure portal first before continuing the tutorial. 
+
+
+## Create a change data capture artifact
+
+1.	Navigate to the **Author** blade in your data factory. You will see a new top-level artifact under **Pipelines** called **Change data capture (preview)**. 
+  
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-2.png" alt-text="Screenshot of new top level artifact shown under Factory resources panel.":::
+
+2.	To create a new **Change data capture**, hover over **Change data capture (preview)** until you see 3 dots appear. Click on the **Change data capture actions**. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-3.png" alt-text="Screenshot of Change data capture (preview) Actions after hovering on the new top-level artifact.":::
+
+3.	Select **New change data capture (preview)**. This will open a flyout to begin the guided process. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-4.png" alt-text="Screenshot of a list of change data capture actions.":::
+  
+4.	You will then be prompted to name your CDC resource. By default, the name will be set to “adfcdc” and continue to increment up by 1. You can replace this default name with your own. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-5.png" alt-text="Screenshot of the text box to update the name of the resource.":::
+
+5.	 Use the drop-down selection list to choose your data source. For this tutorial, we will use **DelimitedText**. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-6.png" alt-text="Screenshot of the guided process flyout with source options in a drop-down selection menu."::: 
+
+6.	You will then be prompted to select a linked service. Create a new linked service or select an existing one. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-7.png" alt-text="Screenshot of the selection box to choose or create a new linked service.":::
+  
+7.	Use the **Browse** button to select your source data folder. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-8.png" alt-text="Screenshot of a folder icon to browse for a folder path.":::
+
+8.	Once you’ve selected a folder path, click **Continue** to set your data target. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-9.png" alt-text="Screenshot of the continue button in the guided process to proceed to select data targets.":::
+
+> [!NOTE]
+> You can choose to add multiple source folders with the **+** button. The other sources must also use the same linked service that you’ve already selected. 
+
+9.	Then, select a **Target type** using the drop-down selection. For this tutorial, we will select **Azure SQL Database**. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-10.png" alt-text="Screenshot of a drop-down selection menu of all data target types.":::
+
+10.	You will then be prompted to select a linked service. Create a new linked service or select an existing one. 
+ 
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-11.png" alt-text="Screenshot of the selection box to choose or create a new linked service to your data target.":::
+ 
+11.	Create new **Target table(s)** or select an existing **Target table(s)**. Use the checkbox to make your selection(s). The **Preview** button will allow you to view your table data. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-12.png" alt-text="Screenshot of the create new tables button and the selection boxes to choose tables for your target.":::
+
+12.	Click **Continue** when you have finalized your selection(s). 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-13.png" alt-text="Screenshot of the continue button in the guided process to proceed to the next step.":::
+
+> [!NOTE]
+> You can choose multiple target tables from your SQL DB. Use the check boxes to select all targets. 
+
+13.	You will automatically land in a new change data capture tab, where you can configure your new resource. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-14.png" alt-text="Screenshot of the change data capture studio.":::
+ 
+14.	A new mapping will automatically be created for you. You can update the **Source** and **Target** selections for your mapping by using the drop-down selection lists. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-15.png" alt-text="Screenshot of the source to target mapping in the change data capture studio.":::
+
+15.	Once you’ve selected your tables, you should see that there are columns mapped. Select the **Column mappings** button to view the column mappings. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-16.png" alt-text="Screenshot of the mapping icon to view column mappings.":::
+
+16.	Here you can view your column mappings. Use the drop-down lists to edit your column mappings for **Mapping method**, **Source column**, and **Target** column.
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-17.png" alt-text="Screenshot of the column mappings.":::
+
+  You can add additional column mappings using the **New mapping** button. Use the drop-down lists to select the **Mapping method**, **Source column**, and **Target** column. 
+  
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-18.png" alt-text="Screenshot of the Add new mapping icon to add new column mappings.":::
+
+17.	When your mapping is complete, click the back arrow to return to the main canvas. 
+
+   :::image type="content" source="media/adf-cdc/change-data-capture-resource-19.png" alt-text="Screenshot of the arrow icon to return to the main change data capture canvas.":::
+
+> [!NOTE] 
+> You can add additional source to target mappings in one CDC artifact. Use the edit button to select more data sources and targets. Then, click **New mapping** and use the drop-down lists to set a new source and target mapping.
+
+   :::image type="content" source="media/adf-cdc/change-data-capture-resource-20.png" alt-text="Screenshot of the edit button to add new sources.":::
+   
+   :::image type="content" source="media/adf-cdc/change-data-capture-resource-21.png" alt-text="Screenshot of the new mapping button to set a new source to target mapping.":::
+
+18.	Once your mapping complete, set your frequency using the **Set Latency** button. 
+
+   :::image type="content" source="media/adf-cdc/change-data-capture-resource-22.png" alt-text="Screenshot of the set frequency button at the top of the canvas.":::
+   
+19.	Select the cadence of your change data capture and click **Apply** to make the changes. By default, it will be set to 15 minutes. 
+
+For example, if you select 30 minutes, every 30 minutes, your change data capture will process your source data and pick up any changed data since the last processed time. 
+
+:::image type="content" source="media/adf-cdc/change-data-capture-resource-23.png" alt-text="Screenshot of the set frequency selection menu.":::
+
+> [!NOTE] 
+> The option to select Real-time to enable streaming data integration is coming soon. 
+
+20.	Once everything has been finalized, publish your changes. 
+
+:::image type="content" source="media/adf-cdc/change-data-capture-resource-24.png" alt-text="Screenshot of the publish button at the top of the canvas.":::
+
+> [!NOTE] 
+> If you do not publish your changes, you will not be able to start your CDC resource. The start button will be grayed out. 
+
+21.	Click **Start** to start running your **Change data capture**. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-25.png" alt-text="Screenshot of the start button at the top of the canvas.":::
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-26.png" alt-text="Screenshot of an actively running change data capture resource.":::
+   
+
+## Monitor your Change data capture
+
+1.	To monitor your change data capture, navigate to the **Monitor** blade or click the monitoring icon from the CDC designer. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-27.png" alt-text="Screenshot of the monitoring blade.":::
+ 
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-28.png" alt-text="Screenshot of the monitoring button at the top of the change data capture canvas.":::
+
+2.	Select **Change data capture** to view your CDC resources. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-29.png" alt-text="Screenshot of the Change data capture monitoring section.":::
+ 
+3.	Here you can see the **Source**, **Target**, **Status**, and **Last processed** time of your change data capture. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-30.png" alt-text="Screenshot of an overview of the change data capture monitoring page.":::
+
+4.	Click the name of your CDC to see more details. You can see how many rows were read and written and other diagnostic information. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-31.png" alt-text="Screenshot of the detailed monitoring of a selected change data capture.":::
+
+> [!NOTE] 
+> If you have multiple mappings set up in your Change data capture, each mapping will show as a different color. Click on the bar to see specific details for each mapping or use the Diagnostics at the bottom of the screen. 
+
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-32.png" alt-text="Screenshot of the detailed monitoring page of a change data capture with multiple sources to target mappings.":::
+  
+  :::image type="content" source="media/adf-cdc/change-data-capture-resource-33.png" alt-text="Screenshot of a detailed breakdown of each mapping in the change data capture artifact.":::
+  
+  
+## Next steps
+- [Learn more about the change data capture resource](concepts-change-data-capture-resource.md)