Skip to content

Commit 09cf21c

Browse files
committed
Fixes
1 parent 9342d04 commit 09cf21c

4 files changed

+84
-55
lines changed

articles/data-factory/v1/data-factory-build-your-first-pipeline-using-arm.md

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -300,24 +300,30 @@ Create a JSON file named **ADFTutorialARM-Parameters.json** that contains parame
300300
```
301301

302302
> [!IMPORTANT]
303-
> You may have separate parameter JSON files for development, testing, and production environments that you can use with the same Data Factory JSON template. By using a Power Shell script, you can automate deploying Data Factory entities in these environments.
304-
>
305-
>
303+
> You may have separate parameter JSON files for development, testing, and production environments that you can use with the same Data Factory JSON template. By using a Power Shell script, you can automate deploying Data Factory entities in these environments.
306304
307305
## Create data factory
308-
1. Start **Azure PowerShell** and run the following command:
306+
307+
1. Start **Azure PowerShell** and run the following command:
308+
309309
* Run the following command and enter the user name and password that you use to sign in to the Azure portal.
310-
```PowerShell
310+
311+
```powershell
311312
Connect-AzAccount
312-
```
313+
```
314+
313315
* Run the following command to view all the subscriptions for this account.
314-
```PowerShell
316+
317+
```powershell
315318
Get-AzSubscription
316-
```
317-
* Run the following command to select the subscription that you want to work with. This subscription should be the same as the one you used in the Azure portal.
318319
```
320+
321+
* Run the following command to select the subscription that you want to work with. This subscription should be the same as the one you used in the Azure portal.
322+
323+
```powershell
319324
Get-AzSubscription -SubscriptionName <SUBSCRIPTION NAME> | Set-AzContext
320-
```
325+
```
326+
321327
2. Run the following command to deploy Data Factory entities using the Resource Manager template you created in Step 1.
322328
323329
```powershell
@@ -560,17 +566,18 @@ You define a pipeline that transform data by running Hive script on an on-demand
560566
```
561567

562568
## Reuse the template
563-
In the tutorial, you created a template for defining Data Factory entities and a template for passing values for parameters. To use the same template to deploy Data Factory entities to different environments, you create a parameter file for each environment and use it when deploying to that environment.
569+
In the tutorial, you created a template for defining Data Factory entities and a template for passing values for parameters. To use the same template to deploy Data Factory entities to different environments, you create a parameter file for each environment and use it when deploying to that environment.
564570

565-
Example:
571+
Example:
566572

567-
```PowerShell
573+
```powershell
568574
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFTutorialARM.json -TemplateParameterFile ADFTutorialARM-Parameters-Dev.json
569575
570576
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFTutorialARM.json -TemplateParameterFile ADFTutorialARM-Parameters-Test.json
571577
572578
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFTutorialARM.json -TemplateParameterFile ADFTutorialARM-Parameters-Production.json
573579
```
580+
574581
Notice that the first command uses parameter file for the development environment, second one for the test environment, and the third one for the production environment.
575582

576583
You can also reuse the template to perform repeated tasks. For example, you need to create many data factories with one or more pipelines that implement the same logic but each data factory uses different Azure storage and Azure SQL Database accounts. In this scenario, you use the same template in the same environment (dev, test, or production) with different parameter files to create data factories.

articles/data-factory/v1/data-factory-build-your-first-pipeline-using-powershell.md

Lines changed: 56 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -43,27 +43,36 @@ The pipeline in this tutorial has one activity: **HDInsight Hive activity**. Thi
4343
In this step, you use Azure PowerShell to create an Azure Data Factory named **FirstDataFactoryPSH**. A data factory can have one or more pipelines. A pipeline can have one or more activities in it. For example, a Copy Activity to copy data from a source to a destination data store and a HDInsight Hive activity to run a Hive script to transform input data. Let's start with creating the data factory in this step.
4444

4545
1. Start Azure PowerShell and run the following command. Keep Azure PowerShell open until the end of this tutorial. If you close and reopen, you need to run these commands again.
46+
4647
* Run the following command and enter the user name and password that you use to sign in to the Azure portal.
47-
```PowerShell
48+
49+
```powershell
4850
Connect-AzAccount
49-
```
51+
```
52+
5053
* Run the following command to view all the subscriptions for this account.
51-
```PowerShell
54+
55+
```powershell
5256
Get-AzSubscription
5357
```
58+
5459
* Run the following command to select the subscription that you want to work with. This subscription should be the same as the one you used in the Azure portal.
55-
```PowerShell
60+
61+
```powershell
5662
Get-AzSubscription -SubscriptionName <SUBSCRIPTION NAME> | Set-AzContext
57-
```
63+
```
64+
5865
2. Create an Azure resource group named **ADFTutorialResourceGroup** by running the following command:
5966
60-
```PowerShell
67+
```powershell
6168
New-AzResourceGroup -Name ADFTutorialResourceGroup -Location "West US"
6269
```
70+
6371
Some of the steps in this tutorial assume that you use the resource group named ADFTutorialResourceGroup. If you use a different resource group, you need to use it in place of ADFTutorialResourceGroup in this tutorial.
72+
6473
3. Run the **New-AzDataFactory** cmdlet that creates a data factory named **FirstDataFactoryPSH**.
6574
66-
```PowerShell
75+
```powershell
6776
New-AzDataFactory -ResourceGroupName ADFTutorialResourceGroup -Name FirstDataFactoryPSH –Location "West US"
6877
```
6978
@@ -76,14 +85,16 @@ Note the following:
7685
7786
* In Azure PowerShell, run the following command to register the Data Factory provider:
7887
79-
```PowerShell
80-
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
88+
```powershell
89+
Register-AzResourceProvider -ProviderNamespace Microsoft.DataFactory
8190
```
82-
You can run the following command to confirm that the Data Factory provider is registered:
8391
84-
```PowerShell
85-
Get-AzResourceProvider
92+
You can run the following command to confirm that the Data Factory provider is registered:
93+
94+
```powershell
95+
Get-AzResourceProvider
8696
```
97+
8798
* Login using the Azure subscription into the [Azure portal](https://portal.azure.com) and navigate to a Data Factory blade (or) create a data factory in the Azure portal. This action automatically registers the provider for you.
8899
89100
Before creating a pipeline, you need to create a few Data Factory entities first. You first create linked services to link data stores/computes to your data store, define input and output datasets to represent input/output data in linked data stores, and then create the pipeline with an activity that uses these datasets.
@@ -112,19 +123,21 @@ In this step, you link your Azure Storage account to your data factory. You use
112123
2. In Azure PowerShell, switch to the ADFGetStarted folder.
113124
3. You can use the **New-AzDataFactoryLinkedService** cmdlet that creates a linked service. This cmdlet and other Data Factory cmdlets you use in this tutorial requires you to pass values for the *ResourceGroupName* and *DataFactoryName* parameters. Alternatively, you can use **Get-AzDataFactory** to get a **DataFactory** object and pass the object without typing *ResourceGroupName* and *DataFactoryName* each time you run a cmdlet. Run the following command to assign the output of the **Get-AzDataFactory** cmdlet to a **$df** variable.
114125
115-
```PowerShell
116-
$df=Get-AzDataFactory -ResourceGroupName ADFTutorialResourceGroup -Name FirstDataFactoryPSH
126+
```powershell
127+
$df = Get-AzDataFactory -ResourceGroupName ADFTutorialResourceGroup -Name FirstDataFactoryPSH
117128
```
118129
4. Now, run the **New-AzDataFactoryLinkedService** cmdlet that creates the linked **StorageLinkedService** service.
119130
120-
```PowerShell
121-
New-AzDataFactoryLinkedService $df -File .\StorageLinkedService.json
131+
```powershell
132+
New-AzDataFactoryLinkedService $df -File .\StorageLinkedService.json
122133
```
134+
123135
If you hadn't run the **Get-AzDataFactory** cmdlet and assigned the output to the **$df** variable, you would have to specify values for the *ResourceGroupName* and *DataFactoryName* parameters as follows.
124136
125-
```PowerShell
126-
New-AzDataFactoryLinkedService -ResourceGroupName ADFTutorialResourceGroup -DataFactoryName FirstDataFactoryPSH -File .\StorageLinkedService.json
137+
```powershell
138+
New-AzDataFactoryLinkedService -ResourceGroupName ADFTutorialResourceGroup -DataFactoryName FirstDataFactoryPSH -File .\StorageLinkedService.json
127139
```
140+
128141
If you close Azure PowerShell in the middle of the tutorial, you have to run the **Get-AzDataFactory** cmdlet next time you start Azure PowerShell to complete the tutorial.
129142
130143
### Create Azure HDInsight linked service
@@ -165,9 +178,9 @@ In this step, you link an on-demand HDInsight cluster to your data factory. The
165178
166179
See [On-demand HDInsight Linked Service](data-factory-compute-linked-services.md#azure-hdinsight-on-demand-linked-service) for details.
167180
2. Run the **New-AzDataFactoryLinkedService** cmdlet that creates the linked service called HDInsightOnDemandLinkedService.
168-
169-
```PowerShell
170-
New-AzDataFactoryLinkedService $df -File .\HDInsightOnDemandLinkedService.json
181+
182+
```powershell
183+
New-AzDataFactoryLinkedService $df -File .\HDInsightOnDemandLinkedService.json
171184
```
172185
173186
## Create datasets
@@ -214,16 +227,16 @@ In this step, you create datasets to represent the input and output data for Hiv
214227
| external |this property is set to true if the input data is not generated by the Data Factory service. |
215228
2. Run the following command in Azure PowerShell to create the Data Factory dataset:
216229
217-
```PowerShell
218-
New-AzDataFactoryDataset $df -File .\InputTable.json
230+
```powershell
231+
New-AzDataFactoryDataset $df -File .\InputTable.json
219232
```
220233
221234
### Create output dataset
222235
Now, you create the output dataset to represent the output data stored in the Azure Blob storage.
223236
224237
1. Create a JSON file named **OutputTable.json** in the **C:\ADFGetStarted** folder with the following content:
225238
226-
```json
239+
```json
227240
{
228241
"name": "AzureBlobOutput",
229242
"properties": {
@@ -243,11 +256,13 @@ Now, you create the output dataset to represent the output data stored in the Az
243256
}
244257
}
245258
```
259+
246260
The JSON defines a dataset named **AzureBlobOutput**, which represents output data for an activity in the pipeline. In addition, it specifies that the results are stored in the blob container called **adfgetstarted** and the folder called **partitioneddata**. The **availability** section specifies that the output dataset is produced on a monthly basis.
261+
247262
2. Run the following command in Azure PowerShell to create the Data Factory dataset:
248263
249-
```PowerShell
250-
New-AzDataFactoryDataset $df -File .\OutputTable.json
264+
```powershell
265+
New-AzDataFactoryDataset $df -File .\OutputTable.json
251266
```
252267
253268
## Create pipeline
@@ -319,27 +334,30 @@ In this step, you create your first pipeline with a **HDInsightHive** activity.
319334
320335
2. Confirm that you see the **input.log** file in the **adfgetstarted/inputdata** folder in the Azure blob storage, and run the following command to deploy the pipeline. Since the **start** and **end** times are set in the past and **isPaused** is set to false, the pipeline (activity in the pipeline) runs immediately after you deploy.
321336
322-
```PowerShell
323-
New-AzDataFactoryPipeline $df -File .\MyFirstPipelinePSH.json
337+
```powershell
338+
New-AzDataFactoryPipeline $df -File .\MyFirstPipelinePSH.json
324339
```
340+
325341
3. Congratulations, you have successfully created your first pipeline using Azure PowerShell!
326342
327343
## Monitor pipeline
328344
In this step, you use Azure PowerShell to monitor what’s going on in an Azure data factory.
329345
330346
1. Run **Get-AzDataFactory** and assign the output to a **$df** variable.
331347
332-
```PowerShell
333-
$df=Get-AzDataFactory -ResourceGroupName ADFTutorialResourceGroup -Name FirstDataFactoryPSH
348+
```powershell
349+
$df = Get-AzDataFactory -ResourceGroupName ADFTutorialResourceGroup -Name FirstDataFactoryPSH
334350
```
351+
335352
2. Run **Get-AzDataFactorySlice** to get details about all slices of the **EmpSQLTable**, which is the output table of the pipeline.
336353
337-
```PowerShell
338-
Get-AzDataFactorySlice $df -DatasetName AzureBlobOutput -StartDateTime 2017-07-01
354+
```powershell
355+
Get-AzDataFactorySlice $df -DatasetName AzureBlobOutput -StartDateTime 2017-07-01
339356
```
357+
340358
Notice that the StartDateTime you specify here is the same start time specified in the pipeline JSON. Here is the sample output:
341359
342-
```PowerShell
360+
```output
343361
ResourceGroupName : ADFTutorialResourceGroup
344362
DataFactoryName : FirstDataFactoryPSH
345363
DatasetName : AzureBlobOutput
@@ -351,15 +369,16 @@ In this step, you use Azure PowerShell to monitor what’s going on in an Azure
351369
LatencyStatus :
352370
LongRetryCount : 0
353371
```
372+
354373
3. Run **Get-AzDataFactoryRun** to get the details of activity runs for a specific slice.
355374
356-
```PowerShell
357-
Get-AzDataFactoryRun $df -DatasetName AzureBlobOutput -StartDateTime 2017-07-01
375+
```powershell
376+
Get-AzDataFactoryRun $df -DatasetName AzureBlobOutput -StartDateTime 2017-07-01
358377
```
359378
360379
Here is the sample output:
361380
362-
```PowerShell
381+
```output
363382
Id : 0f6334f2-d56c-4d48-b427-d4f0fb4ef883_635268096000000000_635292288000000000_AzureBlobOutput
364383
ResourceGroupName : ADFTutorialResourceGroup
365384
DataFactoryName : FirstDataFactoryPSH
@@ -378,6 +397,7 @@ In this step, you use Azure PowerShell to monitor what’s going on in an Azure
378397
PipelineName : MyFirstPipeline
379398
Type : Script
380399
```
400+
381401
You can keep running this cmdlet until you see the slice in **Ready** state or **Failed** state. When the slice is in Ready state, check the **partitioneddata** folder in the **adfgetstarted** container in your blob storage for the output data. Creation of an on-demand HDInsight cluster usually takes some time.
382402
383403
:::image type="content" source="./media/data-factory-build-your-first-pipeline-using-powershell/three-ouptut-files.png" alt-text="output data":::

articles/data-factory/v1/data-factory-copy-activity-tutorial-using-azure-resource-manager-template.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -580,17 +580,19 @@ You define a pipeline that copies data from the Azure blob dataset to the Azure
580580
```
581581

582582
## Reuse the template
583-
In the tutorial, you created a template for defining Data Factory entities and a template for passing values for parameters. The pipeline copies data from an Azure Storage account to Azure SQL Database specified via parameters. To use the same template to deploy Data Factory entities to different environments, you create a parameter file for each environment and use it when deploying to that environment.
583+
In the tutorial, you created a template for defining Data Factory entities and a template for passing values for parameters. The pipeline copies data from an Azure Storage account to Azure SQL Database specified via parameters. To use the same template to deploy Data Factory entities to different environments, you create a parameter file for each environment and use it when deploying to that environment.
584584

585-
Example:
585+
Example:
586586

587-
```PowerShell
587+
```powershell
588588
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFCopyTutorialARM.json -TemplateParameterFile ADFCopyTutorialARM-Parameters-Dev.json
589589
```
590-
```PowerShell
590+
591+
```powershell
591592
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFCopyTutorialARM.json -TemplateParameterFile ADFCopyTutorialARM-Parameters-Test.json
592593
```
593-
```PowerShell
594+
595+
```powershell
594596
New-AzResourceGroupDeployment -Name MyARMDeployment -ResourceGroupName ADFTutorialResourceGroup -TemplateFile ADFCopyTutorialARM.json -TemplateParameterFile ADFCopyTutorialARM-Parameters-Production.json
595597
```
596598

articles/data-factory/v1/data-factory-data-management-gateway.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -456,7 +456,7 @@ To encrypt credentials in the Data Factory Editor, do the following steps:
456456
4. Click **OK** to encrypt credentials and close the dialog box.
457457
8. You should see a **encryptedCredential** property in the **connectionString** now.
458458
459-
```json`
459+
```json
460460
{
461461
"name": "SqlServerLinkedService",
462462
"properties": {

0 commit comments

Comments
 (0)