You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/solution-template-replicate-multiple-objects-sap-cdc.md
+62-13Lines changed: 62 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,29 +17,78 @@ This article describes a solution template that you can use to replicate multipl
17
17
18
18
## About this solution template
19
19
20
-
This template reads an external control file in csv format on your storage store, which contains your SAP ODP contexts, SAP ODP objects and key columns from SAP source system as well as your containers, folders and partitions from Azure Data Lake Gen2 destination store. It then copies each of the SAP ODP object from SAP system to Azure Data Lake Gen2 in Delta format.
20
+
This template reads an external control file in json format on your storage store, which contains your SAP ODP contexts, SAP ODP objects and key columns from SAP source system as well as your containers, folders and partitions from Azure Data Lake Gen2 destination store. It then copies each of the SAP ODP object from SAP system to Azure Data Lake Gen2 in Delta format.
21
21
22
22
The template contains three activities:
23
23
-**Lookup** retrieves the SAP ODP objects list to be loaded and the destination store path from an external control file on your Azure Data Lake Gen2 store.
24
24
-**ForEach** gets the SAP ODP objects list from the Lookup activity and iterates each object to the mapping dataflow activity.
25
25
-**Mapping dataflow** replicates each SAP ODP object from SAP system to Azure Data Lake Gen2 in Delta format. It will do initial full load in the first run and then do incremental load in the subsequent runs automatically. It will merge the changes to Azure Data Lake Gen2 in Delta format.
26
26
27
-
An external control file in csv format is required for in this template. The schema for the control file is as below.
28
-
-*context* is your SAP ODP context from the source SAP system. You can get more details [here](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-the-source-dataset).
29
-
-*object* is your SAP ODP object name to be loaded from the SAP system. You can get more details [here](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-the-source-dataset).
30
-
-*keys* are your key column names from SAP ODP objects used to do the dedupe in mapping dataflow.
31
-
-*container* is your container name in the Azure Data Lake Gen2 as the destination store.
32
-
-*folder* is your folder name in the Azure Data Lake Gen2 as the destination store.
33
-
-*partition* is your column name used to create partitions for each unique value in such column to write data into Delta format on Azure Data Lake Gen2 via Spark cluster used by mapping dataflow. You can get more details [here](concepts-data-flow-performance.md#key)
34
-
35
-
:::image type="content" source="media/solution-template-replicate-multiple-objects-sap-cdc/sap-cdc-template-control-file.png" alt-text="Screenshot of SAP CDC control file.":::
36
-
27
+
An external control file in json format is required in this template. The schema for the control file is as below.
28
+
-*checkPointKey* is your custom key to manage the checkpoint of your changed data capture in ADF. You can get more details [here](concepts-change-data-capture.md#checkpoint).
29
+
-*sapContext* is your SAP ODP context from the source SAP system. You can get more details [here](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-the-source-dataset).
30
+
-*sapObjectName* is your SAP ODP object name to be loaded from the SAP system. You can get more details [here](sap-change-data-capture-prepare-linked-service-source-dataset.md#set-up-the-source-dataset).
31
+
-*sapRunMode* is to determine how you want to load SAP object. It can be fullLoad, incrementalLoad or fullAndIncrementalLoad.
32
+
-*sapKeyColumns* are your key column names from SAP ODP objects used to do the dedupe in mapping dataflow.
33
+
-*sapPartitions* are list of partition conditions leading to separate extraction processes in the connected SAP system.
34
+
-*deltaContainer* is your container name in the Azure Data Lake Gen2 as the destination store.
35
+
-*deltaFolder* is your folder name in the Azure Data Lake Gen2 as the destination store.
36
+
-*deltaKeyColumns* are your columns used to determine if a row from the source matches a row from the sink when you want to update or delete a row.
37
+
-*deltaPartition* is your column used to create partitions for each unique value in such column to write data into Delta format on Azure Data Lake Gen2 via Spark cluster used by mapping dataflow. You can get more details [here](concepts-data-flow-performance.md#key)
1. Create and upload a control file into CSV format to your Azure Data Lake Gen2 as the destination store. The default container to store the control file is **demo** and default control file name is **SAP2DeltaLookup.csv**.
90
+
1. Create and upload a control file into json format to your Azure Data Lake Gen2 as the destination store. The default container to store the control file is **demo** and default control file name is **SapToDeltaParameters.json**.
41
91
42
-
:::image type="content" source="media/solution-template-replicate-multiple-objects-sap-cdc/sap-cdc-template-control-file.png" alt-text="Screenshot of SAP CDC control file.":::
43
92
44
93
2. Go to the **Replicate multiple tables from SAP ODP to Azure Data Lake Storage Gen2 in Delta format** template and **click** it.
0 commit comments