Skip to content

Commit b8f638b

Browse files
committed
Fixes
1 parent 09cf21c commit b8f638b

6 files changed

+106
-85
lines changed

articles/data-factory/tutorial-bulk-copy.md

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,9 @@ For both SQL Database and Azure Synapse Analytics, allow Azure services to acces
8989
2. Run the **Set-AzDataFactoryV2** cmdlet to create a data factory. Replace place-holders with your own values before executing the command.
9090
9191
```powershell
92-
$resourceGroupName = "<your resource group to create the factory>"
93-
$dataFactoryName = "<specify the name of data factory to create. It must be globally unique.>"
94-
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Location "East US" -Name $dataFactoryName
92+
$resourceGroupName = "<your resource group to create the factory>"
93+
$dataFactoryName = "<specify the name of data factory to create. It must be globally unique.>"
94+
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Location "East US" -Name $dataFactoryName
9595
```
9696
9797
Note the following points:
@@ -478,31 +478,31 @@ This pipeline performs two steps:
478478
2. Run the following script to continuously check the run status of pipeline **GetTableListAndTriggerCopyData**, and print out the final pipeline run and activity run result.
479479
480480
```powershell
481-
while ($True) {
481+
while ($True) {
482482
$run = Get-AzDataFactoryV2PipelineRun -ResourceGroupName $resourceGroupName -DataFactoryName $DataFactoryName -PipelineRunId $runId
483483
484484
if ($run) {
485485
if ($run.Status -ne 'InProgress') {
486-
Write-Host "Pipeline run finished. The status is: " $run.Status -foregroundcolor "Yellow"
487-
Write-Host "Pipeline run details:" -foregroundcolor "Yellow"
486+
Write-Host "Pipeline run finished. The status is: " $run.Status -ForegroundColor "Yellow"
487+
Write-Host "Pipeline run details:" -ForegroundColor "Yellow"
488488
$run
489489
break
490490
}
491-
Write-Host "Pipeline is running...status: InProgress" -foregroundcolor "Yellow"
491+
Write-Host "Pipeline is running...status: InProgress" -ForegroundColor "Yellow"
492492
}
493493
494494
Start-Sleep -Seconds 15
495495
}
496496
497497
$result = Get-AzDataFactoryV2ActivityRun -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -PipelineRunId $runId -RunStartedAfter (Get-Date).AddMinutes(-30) -RunStartedBefore (Get-Date).AddMinutes(30)
498-
Write-Host "Activity run details:" -foregroundcolor "Yellow"
498+
Write-Host "Activity run details:" -ForegroundColor "Yellow"
499499
$result
500500
```
501501
502502
Here is the output of the sample run:
503503
504-
```console
505-
Pipeline run details:
504+
```output
505+
Pipeline run details:
506506
ResourceGroupName : <resourceGroupName>
507507
DataFactoryName : <dataFactoryName>
508508
RunId : 0000000000-00000-0000-0000-000000000000
@@ -548,7 +548,7 @@ This pipeline performs two steps:
548548
3. You can get the run ID of pipeline "**IterateAndCopySQLTables**", and check the detailed activity run result as the following.
549549
550550
```powershell
551-
Write-Host "Pipeline 'IterateAndCopySQLTables' run result:" -foregroundcolor "Yellow"
551+
Write-Host "Pipeline 'IterateAndCopySQLTables' run result:" -ForegroundColor "Yellow"
552552
($result | Where-Object {$_.ActivityName -eq "TriggerCopy"}).Output.ToString()
553553
```
554554
@@ -568,6 +568,7 @@ This pipeline performs two steps:
568568
3. Connect to your sink Azure Synapse Analytics and confirm that data has been copied from Azure SQL Database properly.
569569
570570
## Next steps
571+
571572
You performed the following steps in this tutorial:
572573
573574
> [!div class="checklist"]
@@ -579,5 +580,6 @@ You performed the following steps in this tutorial:
579580
> * Monitor the pipeline and activity runs.
580581
581582
Advance to the following tutorial to learn about copy data incrementally from a source to a destination:
583+
582584
> [!div class="nextstepaction"]
583-
>[Copy data incrementally](tutorial-incremental-copy-powershell.md)
585+
> [Copy data incrementally](tutorial-incremental-copy-powershell.md)

articles/data-factory/tutorial-incremental-copy-portal.md

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -65,23 +65,24 @@ If you don't have an Azure subscription, create a [free](https://azure.microsoft
6565
2. Run the following SQL command against your SQL database to create a table named `data_source_table` as the data source store:
6666

6767
```sql
68-
create table data_source_table
69-
(
70-
PersonID int,
71-
Name varchar(255),
72-
LastModifytime datetime
73-
);
74-
75-
INSERT INTO data_source_table
76-
(PersonID, Name, LastModifytime)
77-
VALUES
78-
(1, 'aaaa','9/1/2017 12:56:00 AM'),
79-
(2, 'bbbb','9/2/2017 5:23:00 AM'),
80-
(3, 'cccc','9/3/2017 2:36:00 AM'),
81-
(4, 'dddd','9/4/2017 3:21:00 AM'),
82-
(5, 'eeee','9/5/2017 8:06:00 AM');
68+
create table data_source_table
69+
(
70+
PersonID int,
71+
Name varchar(255),
72+
LastModifytime datetime
73+
);
74+
75+
INSERT INTO data_source_table
76+
(PersonID, Name, LastModifytime)
77+
VALUES
78+
(1, 'aaaa','9/1/2017 12:56:00 AM'),
79+
(2, 'bbbb','9/2/2017 5:23:00 AM'),
80+
(3, 'cccc','9/3/2017 2:36:00 AM'),
81+
(4, 'dddd','9/4/2017 3:21:00 AM'),
82+
(5, 'eeee','9/5/2017 8:06:00 AM');
8383
```
84-
In this tutorial, you use LastModifytime as the watermark column. The data in the data source store is shown in the following table:
84+
85+
In this tutorial, you use LastModifytime as the watermark column. The data in the data source store is shown in the following table:
8586

8687
```
8788
PersonID | Name | LastModifytime

articles/data-factory/v1/data-factory-build-your-first-pipeline-using-powershell.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ In this step, you link your Azure Storage account to your data factory. You use
107107
108108
1. Create a JSON file named StorageLinkedService.json in the C:\ADFGetStarted folder with the following content. Create the folder ADFGetStarted if it does not already exist.
109109
110-
```json
110+
```json
111111
{
112112
"name": "StorageLinkedService",
113113
"properties": {
@@ -145,7 +145,7 @@ In this step, you link an on-demand HDInsight cluster to your data factory. The
145145
146146
1. Create a JSON file named **HDInsightOnDemandLinkedService**.json in the **C:\ADFGetStarted** folder with the following content.
147147
148-
```json
148+
```json
149149
{
150150
"name": "HDInsightOnDemandLinkedService",
151151
"properties": {

articles/data-factory/v1/data-factory-build-your-first-pipeline-using-rest-api.md

Lines changed: 32 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -291,7 +291,7 @@ In this step, you create an Azure Data Factory named **FirstDataFactoryREST**. A
291291

292292
Confirm that the name of the data factory you specify here (ADFCopyTutorialDF) matches the name specified in the **datafactory.json**.
293293

294-
```powershell
294+
```powershell
295295
$cmd = {.\curl.exe -X PUT -H "Authorization: Bearer $accessToken" -H "Content-Type: application/json" --data "@datafactory.json" https://management.azure.com/subscriptions/$subscription_id/resourcegroups/$rg/providers/Microsoft.DataFactory/datafactories/FirstDataFactoryREST?api-version=2015-10-01};
296296
```
297297
2. Run the command by using **Invoke-Command**.
@@ -317,14 +317,16 @@ Note the following points:
317317
318318
* In Azure PowerShell, run the following command to register the Data Factory provider:
319319
320-
```powershell
321-
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
322-
```
320+
```powershell
321+
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
322+
```
323+
324+
You can run the following command to confirm that the Data Factory provider is registered:
325+
326+
```powershell
327+
Get-AzResourceProvider
328+
```
323329
324-
You can run the following command to confirm that the Data Factory provider is registered:
325-
```powershell
326-
Get-AzResourceProvider
327-
```
328330
* Login using the Azure subscription into the [Azure portal](https://portal.azure.com) and navigate to a Data Factory blade (or) create a data factory in the Azure portal. This action automatically registers the provider for you.
329331
330332
Before creating a pipeline, you need to create a few Data Factory entities first. You first create linked services to link data stores/computes to your data store, define input and output datasets to represent data in linked data stores.
@@ -337,38 +339,42 @@ In this step, you link your Azure Storage account to your data factory. With thi
337339
338340
1. Assign the command to variable named **cmd**.
339341
340-
```powershell
341-
$cmd = {.\curl.exe -X PUT -H "Authorization: Bearer $accessToken" -H "Content-Type: application/json" --data “@azurestoragelinkedservice.json” https://management.azure.com/subscriptions/$subscription_id/resourcegroups/$rg/providers/Microsoft.DataFactory/datafactories/$adf/linkedservices/AzureStorageLinkedService?api-version=2015-10-01};
342-
```
342+
```powershell
343+
$cmd = { .\curl.exe -X PUT -H "Authorization: Bearer $accessToken" -H "Content-Type: application/json" --data “@azurestoragelinkedservice.json” https://management.azure.com/subscriptions/$subscription_id/resourcegroups/$rg/providers/Microsoft.DataFactory/datafactories/$adf/linkedservices/AzureStorageLinkedService?api-version=2015-10-01 };
344+
```
345+
343346
2. Run the command by using **Invoke-Command**.
344347
345-
```powershell
346-
$results = Invoke-Command -scriptblock $cmd;
347-
```
348+
```powershell
349+
$results = Invoke-Command -scriptblock $cmd;
350+
```
351+
348352
3. View the results. If the linked service has been successfully created, you see the JSON for the linked service in the **results**; otherwise, you see an error message.
349353
350-
```powershell
351-
Write-Host $results
352-
```
354+
```powershell
355+
Write-Host $results
356+
```
353357
354358
### Create Azure HDInsight linked service
355359
In this step, you link an on-demand HDInsight cluster to your data factory. The HDInsight cluster is automatically created at runtime and deleted after it is done processing and idle for the specified amount of time. You could use your own HDInsight cluster instead of using an on-demand HDInsight cluster. See [Compute Linked Services](data-factory-compute-linked-services.md) for details.
356360
357361
1. Assign the command to variable named **cmd**.
358362
359-
```powershell
360-
$cmd = {.\curl.exe -X PUT -H "Authorization: Bearer $accessToken" -H "Content-Type: application/json" --data "@hdinsightondemandlinkedservice.json" https://management.azure.com/subscriptions/$subscription_id/resourcegroups/$rg/providers/Microsoft.DataFactory/datafactories/$adf/linkedservices/hdinsightondemandlinkedservice?api-version=2015-10-01};
361-
```
363+
```powershell
364+
$cmd = {.\curl.exe -X PUT -H "Authorization: Bearer $accessToken" -H "Content-Type: application/json" --data "@hdinsightondemandlinkedservice.json" https://management.azure.com/subscriptions/$subscription_id/resourcegroups/$rg/providers/Microsoft.DataFactory/datafactories/$adf/linkedservices/hdinsightondemandlinkedservice?api-version=2015-10-01};
365+
```
366+
362367
2. Run the command by using **Invoke-Command**.
363368
364-
```powershell
365-
$results = Invoke-Command -scriptblock $cmd;
366-
```
369+
```powershell
370+
$results = Invoke-Command -scriptblock $cmd;
371+
```
372+
367373
3. View the results. If the linked service has been successfully created, you see the JSON for the linked service in the **results**; otherwise, you see an error message.
368374
369-
```powershell
370-
Write-Host $results
371-
```
375+
```powershell
376+
Write-Host $results
377+
```
372378
373379
## Create datasets
374380
In this step, you create datasets to represent the input and output data for Hive processing. These datasets refer to the **StorageLinkedService** you have created earlier in this tutorial. The linked service points to an Azure Storage account and datasets specify container, folder, file name in the storage that holds input and output data.

articles/data-factory/v1/data-factory-build-your-first-pipeline-using-vs.md

Lines changed: 21 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -294,17 +294,17 @@ In this step, you publish the Data Factory entities (linked services, datasets,
294294

295295
:::image type="content" source="media/data-factory-build-your-first-pipeline-using-vs/publish-new-data-factory.png" alt-text="Publish - New data factory settings":::
296296

297-
1. select **Create New Data Factory** option.
297+
1. Select **Create New Data Factory** option.
298298
2. Enter a unique **name** for the data factory. For example: **DataFactoryUsingVS09152016**. The name must be globally unique.
299299
3. Select the right subscription for the **Subscription** field.
300-
> [!IMPORTANT]
301-
> If you do not see any subscription, ensure that you logged in using an account that is an admin or co-admin of the subscription.
300+
> [!IMPORTANT]
301+
> If you do not see any subscription, ensure that you logged in using an account that is an admin or co-admin of the subscription.
302302
4. Select the **resource group** for the data factory to be created.
303303
5. Select the **region** for the data factory.
304304
6. Click **Next** to switch to the **Publish Items** page. (Press **TAB** to move out of the Name field to if the **Next** button is disabled.)
305-
306305
> [!IMPORTANT]
307306
> If you receive the error **Data factory name “DataFactoryUsingVS” is not available** when publishing, change the name (for example, yournameDataFactoryUsingVS). See [Data Factory - Naming Rules](data-factory-naming-rules.md) topic for naming rules for Data Factory artifacts.
307+
308308
1. In the **Publish Items** page, ensure that all the Data Factories entities are selected, and click **Next** to switch to the **Summary** page.
309309

310310
:::image type="content" source="media/data-factory-build-your-first-pipeline-using-vs/publish-items-page.png" alt-text="Publish items page":::
@@ -316,20 +316,27 @@ In this step, you publish the Data Factory entities (linked services, datasets,
316316
Important points to note:
317317

318318
- If you receive the error: **This subscription is not registered to use namespace Microsoft.DataFactory**, do one of the following and try publishing again:
319-
- In Azure PowerShell, run the following command to register the Data Factory provider.
320-
```powershell
321-
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
322-
```
323-
You can run the following command to confirm that the Data Factory provider is registered.
324-
325-
```powershell
326-
Get-AzResourceProvider
327-
```
328-
- Login using the Azure subscription in to the [Azure portal](https://portal.azure.com) and navigate to a Data Factory blade (or) create a data factory in the Azure portal. This action automatically registers the provider for you.
319+
320+
- In Azure PowerShell, run the following command to register the Data Factory provider.
321+
322+
```powershell
323+
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
324+
```
325+
326+
You can run the following command to confirm that the Data Factory provider is registered.
327+
328+
```powershell
329+
Get-AzResourceProvider
330+
```
331+
332+
- Login using the Azure subscription in to the [Azure portal](https://portal.azure.com) and navigate to a Data Factory blade (or) create a data factory in the Azure portal. This action automatically registers the provider for you.
333+
329334
- The name of the data factory may be registered as a DNS name in the future and hence become publicly visible.
335+
330336
- To create Data Factory instances, you need to be an admin or co-admin of the Azure subscription
331337

332338
### Monitor pipeline
339+
333340
In this step, you monitor the pipeline using Diagram View of the data factory.
334341

335342
#### Monitor pipeline using Diagram View

articles/data-factory/v1/data-factory-monitor-manage-pipelines.md

Lines changed: 20 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -209,19 +209,22 @@ If the activity run fails in a pipeline, the dataset that is produced by the pip
209209
:::image type="content" source="./media/data-factory-monitor-manage-pipelines/activity-run-details-with-error.png" alt-text="Activity run details blade with error":::
210210

211211
#### Use PowerShell to debug an error
212+
212213
1. Launch **PowerShell**.
213-
2. Run the **Get-AzDataFactorySlice** command to see the slices and their statuses. You should see a slice with the status of **Failed**.
214+
2. Run the **Get-AzDataFactorySlice** command to see the slices and their statuses. You should see a slice with the status of **Failed**.
214215

215-
```powershell
216-
Get-AzDataFactorySlice [-ResourceGroupName] <String> [-DataFactoryName] <String> [-DatasetName] <String> [-StartDateTime] <DateTime> [[-EndDateTime] <DateTime> ] [-Profile <AzureProfile> ] [ <CommonParameters>]
217-
```
218-
For example:
216+
```powershell
217+
Get-AzDataFactorySlice [-ResourceGroupName] <String> [-DataFactoryName] <String> [-DatasetName] <String> [-StartDateTime] <DateTime> [[-EndDateTime] <DateTime> ] [-Profile <AzureProfile> ] [ <CommonParameters>]
218+
```
219219
220-
```powershell
221-
Get-AzDataFactorySlice -ResourceGroupName ADF -DataFactoryName LogProcessingFactory -DatasetName EnrichedGameEventsTable -StartDateTime 2014-05-04 20:00:00
222-
```
220+
For example:
221+
222+
```powershell
223+
Get-AzDataFactorySlice -ResourceGroupName ADF -DataFactoryName LogProcessingFactory -DatasetName EnrichedGameEventsTable -StartDateTime 2014-05-04 20:00:00
224+
```
223225
224226
Replace **StartDateTime** with start time of your pipeline.
227+
225228
3. Now, run the **Get-AzDataFactoryRun** cmdlet to get details about the activity run for the slice.
226229
227230
```powershell
@@ -231,14 +234,15 @@ If the activity run fails in a pipeline, the dataset that is produced by the pip
231234
232235
For example:
233236
234-
```powershell
237+
```powershell
235238
Get-AzDataFactoryRun -ResourceGroupName ADF -DataFactoryName LogProcessingFactory -DatasetName EnrichedGameEventsTable -StartDateTime "5/5/2014 12:00:00 AM"
236-
```
239+
```
237240
238241
The value of StartDateTime is the start time for the error/problem slice that you noted from the previous step. The date-time should be enclosed in double quotes.
242+
239243
4. You should see output with details about the error that is similar to the following:
240244
241-
```
245+
```output
242246
Id : 841b77c9-d56c-48d1-99a3-8c16c3e77d39
243247
ResourceGroupName : ADF
244248
DataFactoryName : LogProcessingFactory3
@@ -259,12 +263,13 @@ If the activity run fails in a pipeline, the dataset that is produced by the pip
259263
ActivityName : PigEnrichLogs
260264
PipelineName : EnrichGameLogsPipeline
261265
Type :
262-
```
266+
```
267+
263268
5. You can run the **Save-AzDataFactoryLog** cmdlet with the Id value that you see from the output, and download the log files by using the **-DownloadLogsoption** for the cmdlet.
264269
265-
```powershell
266-
Save-AzDataFactoryLog -ResourceGroupName "ADF" -DataFactoryName "LogProcessingFactory" -Id "841b77c9-d56c-48d1-99a3-8c16c3e77d39" -DownloadLogs -Output "C:\Test"
267-
```
270+
```powershell
271+
Save-AzDataFactoryLog -ResourceGroupName "ADF" -DataFactoryName "LogProcessingFactory" -Id "841b77c9-d56c-48d1-99a3-8c16c3e77d39" -DownloadLogs -Output "C:\Test"
272+
```
268273
269274
## Rerun failures in a pipeline
270275

0 commit comments

Comments
 (0)