You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/tutorial-copy-data-portal.md
+42-16Lines changed: 42 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,16 @@
1
1
---
2
-
title: Use the Azure portal to create a data factory pipeline
3
-
description: This tutorial provides step-by-step instructions for using the Azure portal to create a data factory with a pipeline. The pipeline uses the copy activity to copy data from Azure Blob storage to Azure SQL Database.
2
+
title: 'Use the Azure portal to create a data factory pipeline'
3
+
description: This tutorial provides instructions to create a data factory with a pipeline with a copy activity to copy data from Azure Blob storage to Azure SQL Database.
4
4
author: jianleishen
5
5
ms.topic: tutorial
6
6
ms.date: 04/25/2025
7
7
ms.subservice: data-movement
8
8
ms.author: jianleishen
9
+
10
+
#customer intent: As a new Azure Data Factory user I want to create a data factory and quickly create my first pipeline to move data between resources, so I can apply it to my own needs.
9
11
---
10
12
11
-
# Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory
13
+
# Tutorial: Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory
@@ -20,12 +22,13 @@ In this tutorial, you create a data factory by using the Azure Data Factory user
20
22
In this tutorial, you perform the following steps:
21
23
22
24
> [!div class="checklist"]
23
-
> * Create a data factory.
24
-
> * Create a pipeline with a copy activity.
25
+
> *[Create a data factory.](#create-a-data-factory)
26
+
> *[Create a pipeline with a copy activity.](#create-a-pipeline)
25
27
> * Test run the pipeline.
26
-
> * Trigger the pipeline manually.
27
-
> * Trigger the pipeline on a schedule.
28
+
> *[Trigger the pipeline manually.](#trigger-the-pipeline-manually)
29
+
> *[Trigger the pipeline on a schedule.](#trigger-the-pipeline-on-a-schedule)
28
30
> * Monitor the pipeline and activity runs.
31
+
> *[Disable or delete your scheduled trigger.](#disable-trigger)
29
32
30
33
## Prerequisites
31
34
@@ -66,7 +69,7 @@ Now, prepare your Blob storage and SQL database for the tutorial by performing t
66
69
CREATE CLUSTERED INDEX IX_emp_ID ON dbo.emp (ID);
67
70
```
68
71
69
-
1. Allow Azure services to access SQL Server. Ensure that **Allow access to Azure services** is turned **ON** for your SQL Server so that Data Factory can write data to your SQL Server. To verify and turn on this setting, go to your SQL Server in the Azure portal, select **Security** > **Networking** > enable **Selected networks**> chech **Allow Azure services and resources to access this server** under the **Exceptions**.
72
+
1. Allow Azure services to access SQL Server. Ensure that **Allow access to Azure services** is turned **ON** for your SQL Server so that Data Factory can write data to your SQL Server. To verify and turn on this setting, go to your SQL Server in the Azure portal, select **Security** > **Networking** > enable **Selected networks**> check **Allow Azure services and resources to access this server** under the **Exceptions**.
70
73
71
74
## Create a data factory
72
75
@@ -110,7 +113,7 @@ In this step, you create a pipeline with a copy activity in the data factory. Th
110
113
### Configure source
111
114
112
115
>[!TIP]
113
-
>In this tutorial, you use *Account key* as the authentication type for your source data store, but you can choose other supported authentication methods: *SAS URI*,*Service Principal* and *Managed Identity* if needed. Refer to corresponding sections in [this article](./connector-azure-blob-storage.md#linked-service-properties) for details.
116
+
>In this tutorial, you use *Account key* as the authentication type for your source data store, but you can choose other supported authentication methods: *SAS URI*,*Service Principal*, and *Managed Identity* if needed. Refer to corresponding sections in [this article](./connector-azure-blob-storage.md#linked-service-properties) for details.
114
117
>To store secrets for data stores securely, it's also recommended to use an Azure Key Vault. Refer to [this article](./store-credentials-in-key-vault.md) for detailed illustrations.
115
118
116
119
1. Go to the **Source** tab. Select **+ New** to create a source dataset.
@@ -161,7 +164,7 @@ In this step, you create a pipeline with a copy activity in the data factory. Th
161
164
162
165
:::image type="content" source="./media/tutorial-copy-data-portal/new-azure-sql-linked-service-window.png" alt-text="Save new linked service":::
163
166
164
-
1. It automatically navigates to the **Set Properties** dialog box. In **Table**, select **[dbo].[emp]**. Then select **OK**.
167
+
1. It automatically navigates to the **Set Properties** dialog box. In **Table**, select **Enter manually**, and enter **[dbo].[emp]**. Then select **OK**.
165
168
166
169
1. Go to the tab with the pipeline, and in **Sink Dataset**, confirm that **OutputSqlDataset** is selected.
167
170
@@ -170,42 +173,49 @@ In this step, you create a pipeline with a copy activity in the data factory. Th
170
173
You can optionally map the schema of the source to corresponding schema of destination by following [Schema mapping in copy activity](copy-activity-schema-and-type-mapping.md).
171
174
172
175
## Validate the pipeline
176
+
173
177
To validate the pipeline, select **Validate** from the tool bar.
174
178
175
179
You can see the JSON code associated with the pipeline by clicking **Code** on the upper right.
176
180
177
181
## Debug and publish the pipeline
182
+
178
183
You can debug a pipeline before you publish artifacts (linked services, datasets, and pipeline) to Data Factory or your own Azure Repos Git repository.
179
184
180
185
1. To debug the pipeline, select **Debug** on the toolbar. You see the status of the pipeline run in the **Output** tab at the bottom of the window.
181
186
182
187
1. Once the pipeline can run successfully, in the top toolbar, select **Publish all**. This action publishes entities (datasets, and pipelines) you created to Data Factory.
183
188
184
-
1. Wait until you see the **Successfully published** message. To see notification messages, click the **Show Notifications** on the top-right (bell button).
189
+
1. Wait until you see the **Successfully published** notification message. To see notification messages, select the **Show Notifications** on the top-right (bell button).
185
190
186
191
## Trigger the pipeline manually
192
+
187
193
In this step, you manually trigger the pipeline you published in the previous step.
188
194
189
-
1. Select **Trigger** on the toolbar, and then select **Trigger Now**. On the **Pipeline Run** page, select **OK**.
195
+
1. Select **Add trigger** on the toolbar, and then select **Trigger Now**.
196
+
197
+
1. On the **Pipeline Run** page, select **OK**.
190
198
191
199
1. Go to the **Monitor** tab on the left. You see a pipeline run that is triggered by a manual trigger. You can use links under the **PIPELINE NAME** column to view activity details and to rerun the pipeline.
1. To see activity runs associated with the pipeline run, select the **CopyPipeline** link under the **PIPELINE NAME** column. In this example, there's only one activity, so you see only one entry in the list. For details about the copy operation, select the **Details** link (eyeglasses icon) under the **ACTIVITY NAME** column. Select **All pipeline runs** at the top to go back to the Pipeline Runs view. To refresh the view, select **Refresh**.
203
+
1. To see activity runs associated with the pipeline run, select the **CopyPipeline** link under the **PIPELINE NAME** column. In this example, there's only one activity, so you see only one entry in the list. For details about the copy operation, hover over the activity and
204
+
1. select the **Details** link (eyeglasses icon) under the **ACTIVITY NAME** column. Select **All pipeline runs** at the top to go back to the Pipeline Runs view. To refresh the view, select **Refresh**.
1. Verify that two more rows are added to the **emp** table in the database.
200
209
201
210
## Trigger the pipeline on a schedule
211
+
202
212
In this schedule, you create a schedule trigger for the pipeline. The trigger runs the pipeline on the specified schedule, such as hourly or daily. Here you set the trigger to run every minute until the specified end datetime.
203
213
204
214
1. Go to the **Author** tab on the left above the monitor tab.
205
215
206
-
1. Go to your pipeline, click **Trigger** on the tool bar, and select **New/Edit**.
216
+
1. Go to your pipeline, select **Trigger** on the tool bar, and select **New/Edit**.
207
217
208
-
1. In the **Add triggers** dialog box, select **+ New** for **Choose trigger** area.
218
+
1. In the **Add triggers** dialog box, select **Choose trigger** and select **+ New**.
209
219
210
220
1. In the **New Trigger** window, take the following steps:
211
221
@@ -228,7 +238,7 @@ In this schedule, you create a schedule trigger for the pipeline. The trigger ru
228
238
229
239
1. On the **Edit trigger** page, review the warning, and then select **Save**. The pipeline in this example doesn't take any parameters.
230
240
231
-
1. Click **Publish all** to publish the change.
241
+
1. Select **Publish all** to publish the change.
232
242
233
243
1. Go to the **Monitor** tab on the left to see the triggered pipeline runs.
234
244
@@ -240,7 +250,22 @@ In this schedule, you create a schedule trigger for the pipeline. The trigger ru
240
250
241
251
1. Verify that two rows per minute (for each pipeline run) are inserted into the **emp** table until the specified end time.
242
252
253
+
## Disable trigger
254
+
255
+
To disable your every minute trigger that you created, follow these steps:
256
+
257
+
1. Select the **Manage** pane on the left side.
258
+
259
+
1. Under **Author** select **Triggers**.
260
+
261
+
1. Hover over the **RunEveryMinute** trigger you created.
262
+
1. Select the **Stop** button to disable the trigger from running.
263
+
1. Select the **Delete** button to disable and delete the trigger.
264
+
265
+
1. Select **Publish all** to save your changes.
266
+
243
267
## Related content
268
+
244
269
The pipeline in this sample copies data from one location to another location in Blob storage. You learned how to:
245
270
246
271
> [!div class="checklist"]
@@ -250,6 +275,7 @@ The pipeline in this sample copies data from one location to another location in
250
275
> * Trigger the pipeline manually.
251
276
> * Trigger the pipeline on a schedule.
252
277
> * Monitor the pipeline and activity runs.
278
+
> * Disable or delete your scheduled trigger.
253
279
254
280
255
281
Advance to the following tutorial to learn how to copy data from on-premises to the cloud:
0 commit comments