You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/transform-data-using-hadoop-hive.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ author: nabhishek
10
10
ms.author: abnarain
11
11
manager: anandsub
12
12
ms.custom: seo-lt-2019
13
-
ms.date: 01/15/2019
13
+
ms.date: 05/08/2019
14
14
---
15
15
16
16
# Transform data using Hadoop Hive activity in Azure Data Factory
@@ -59,7 +59,7 @@ If you are new to Azure Data Factory, read through [Introduction to Azure Data F
59
59
| description | Text describing what the activity is used for | No |
60
60
| type | For Hive Activity, the activity type is HDinsightHive | Yes |
61
61
| linkedServiceName | Reference to the HDInsight cluster registered as a linked service in Data Factory. To learn about this linked service, see [Compute linked services](compute-linked-services.md) article. | Yes |
62
-
| scriptLinkedService | Reference to an Azure Storage Linked Service used to store the Hive script to be executed. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
62
+
| scriptLinkedService | Reference to an Azure Storage Linked Service used to store the Hive script to be executed. Only **[Azure Blob Storage](https://docs.microsoft.com/azure/data-factory/connector-azure-blob-storage)** and **[ADLS Gen2](https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-storage)** linked services are supported here. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used.| No |
63
63
| scriptPath | Provide the path to the script file stored in the Azure Storage referred by scriptLinkedService. The file name is case-sensitive. | Yes |
64
64
| getDebugInfo | Specifies when the log files are copied to the Azure Storage used by HDInsight cluster (or) specified by scriptLinkedService. Allowed values: None, Always, or Failure. Default value: None. | No |
65
65
| arguments | Specifies an array of arguments for a Hadoop job. The arguments are passed as command-line arguments to each task. | No |
Copy file name to clipboardExpand all lines: articles/data-factory/transform-data-using-hadoop-map-reduce.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ author: nabhishek
9
9
ms.author: abnarain
10
10
manager: shwang
11
11
ms.custom: seo-lt-2019
12
-
ms.date: 01/16/2018
12
+
ms.date: 05/08/2020
13
13
---
14
14
15
15
# Transform data using Hadoop MapReduce activity in Azure Data Factory
@@ -64,7 +64,7 @@ See [Pig](transform-data-using-hadoop-pig.md) and [Hive](transform-data-using-ha
64
64
| type | For MapReduce Activity, the activity type is HDinsightMapReduce | Yes |
65
65
| linkedServiceName | Reference to the HDInsight cluster registered as a linked service in Data Factory. To learn about this linked service, see [Compute linked services](compute-linked-services.md) article. | Yes |
66
66
| className | Name of the Class to be executed | Yes |
67
-
| jarLinkedService | Reference to an Azure Storage Linked Service used to store the Jar files. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
67
+
| jarLinkedService | Reference to an Azure Storage Linked Service used to store the Jar files. Only **[Azure Blob Storage](https://docs.microsoft.com/azure/data-factory/connector-azure-blob-storage)** and **[ADLS Gen2](https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-storage)** linked services are supported here. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
68
68
| jarFilePath | Provide the path to the Jar files stored in the Azure Storage referred by jarLinkedService. The file name is case-sensitive. | Yes |
69
69
| jarlibs | String array of the path to the Jar library files referenced by the job stored in the Azure Storage defined in jarLinkedService. The file name is case-sensitive. | No |
70
70
| getDebugInfo | Specifies when the log files are copied to the Azure Storage used by HDInsight cluster (or) specified by jarLinkedService. Allowed values: None, Always, or Failure. Default value: None. | No |
@@ -107,7 +107,7 @@ You can use the HDInsight MapReduce Activity to run any MapReduce jar file on an
107
107
}
108
108
}
109
109
```
110
-
You can specify any arguments for the MapReduce program in the **arguments** section. At runtime, you see a few extra arguments (for example: mapreduce.job.tags) from the MapReduce framework. To differentiate your arguments with the MapReduce arguments, consider using both option and value as arguments as shown in the following example (-s,--input,--output etc., are options immediately followed by their values).
110
+
You can specify any arguments for the MapReduce program in the **arguments** section. At runtime, you see a few extra arguments (for example: mapreduce.job.tags) from the MapReduce framework. To differentiate your arguments with the MapReduce arguments, consider using both option and value as arguments as shown in the following example (-s,--input,--output etc., are options immediately followed by their values).
111
111
112
112
## Next steps
113
113
See the following articles that explain how to transform data in other ways:
Copy file name to clipboardExpand all lines: articles/data-factory/transform-data-using-hadoop-pig.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ author: nabhishek
10
10
ms.author: abnarain
11
11
manager: shwang
12
12
ms.custom: seo-lt-2019
13
-
ms.date: 01/16/2018
13
+
ms.date: 05/08/2020
14
14
---
15
15
16
16
# Transform data using Hadoop Pig activity in Azure Data Factory
@@ -60,7 +60,7 @@ If you are new to Azure Data Factory, read through [Introduction to Azure Data F
60
60
| description | Text describing what the activity is used for | No |
61
61
| type | For Hive Activity, the activity type is HDinsightPig | Yes |
62
62
| linkedServiceName | Reference to the HDInsight cluster registered as a linked service in Data Factory. To learn about this linked service, see [Compute linked services](compute-linked-services.md) article. | Yes |
63
-
| scriptLinkedService | Reference to an Azure Storage Linked Service used to store the Pig script to be executed. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
63
+
| scriptLinkedService | Reference to an Azure Storage Linked Service used to store the Pig script to be executed. Only **[Azure Blob Storage](https://docs.microsoft.com/azure/data-factory/connector-azure-blob-storage)** and **[ADLS Gen2](https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-storage)** linked services are supported here. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
64
64
| scriptPath | Provide the path to the script file stored in the Azure Storage referred by scriptLinkedService. The file name is case-sensitive. | No |
65
65
| getDebugInfo | Specifies when the log files are copied to the Azure Storage used by HDInsight cluster (or) specified by scriptLinkedService. Allowed values: None, Always, or Failure. Default value: None. | No |
66
66
| arguments | Specifies an array of arguments for a Hadoop job. The arguments are passed as command-line arguments to each task. | No |
Copy file name to clipboardExpand all lines: articles/data-factory/transform-data-using-hadoop-streaming.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ ms.service: data-factory
9
9
ms.workload: data-services
10
10
ms.topic: conceptual
11
11
ms.custom: seo-lt-2019
12
-
ms.date: 01/16/2018
12
+
ms.date: 05/08/2020
13
13
---
14
14
15
15
# Transform data using Hadoop Streaming activity in Azure Data Factory
@@ -73,7 +73,7 @@ If you are new to Azure Data Factory, read through [Introduction to Azure Data F
73
73
| mapper | Specifies the name of the mapper executable | Yes |
74
74
| reducer | Specifies the name of the reducer executable | Yes |
75
75
| combiner | Specifies the name of the combiner executable | No |
76
-
| fileLinkedService | Reference to an Azure Storage Linked Service used to store the Mapper, Combiner, and Reducer programs to be executed. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
76
+
| fileLinkedService | Reference to an Azure Storage Linked Service used to store the Mapper, Combiner, and Reducer programs to be executed. Only **[Azure Blob Storage](https://docs.microsoft.com/azure/data-factory/connector-azure-blob-storage)** and **[ADLS Gen2](https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-storage)** linked services are supported here. If you don't specify this Linked Service, the Azure Storage Linked Service defined in the HDInsight Linked Service is used. | No |
77
77
| filePath | Provide an array of path to the Mapper, Combiner, and Reducer programs stored in the Azure Storage referred by fileLinkedService. The path is case-sensitive. | Yes |
78
78
| input | Specifies the WASB path to the input file for the Mapper. | Yes |
79
79
| output | Specifies the WASB path to the output file for the Reducer. | Yes |
Copy file name to clipboardExpand all lines: articles/data-factory/transform-data-using-spark.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ author: nabhishek
9
9
ms.author: abnarain
10
10
manager: shwang
11
11
ms.custom: seo-lt-2019
12
-
ms.date: 05/31/2018
12
+
ms.date: 05/08/2020
13
13
---
14
14
15
15
# Transform data using Spark activity in Azure Data Factory
@@ -60,7 +60,7 @@ The following table describes the JSON properties used in the JSON definition:
60
60
| description | Text describing what the activity does. | No |
61
61
| type | For Spark Activity, the activity type is HDInsightSpark. | Yes |
62
62
| linkedServiceName | Name of the HDInsight Spark Linked Service on which the Spark program runs. To learn about this linked service, see [Compute linked services](compute-linked-services.md) article. | Yes |
63
-
| SparkJobLinkedService | The Azure Storage linked service that holds the Spark job file, dependencies, and logs. If you do not specify a value for this property, the storage associated with HDInsight cluster is used. The value of this property can only be an Azure Storage linked service. | No |
63
+
| SparkJobLinkedService | The Azure Storage linked service that holds the Spark job file, dependencies, and logs. Only **[Azure Blob Storage](https://docs.microsoft.com/azure/data-factory/connector-azure-blob-storage)** and **[ADLS Gen2](https://docs.microsoft.com/azure/data-factory/connector-azure-data-lake-storage)** linked services are supported here. If you do not specify a value for this property, the storage associated with HDInsight cluster is used. The value of this property can only be an Azure Storage linked service. | No |
64
64
| rootPath | The Azure Blob container and folder that contains the Spark file. The file name is case-sensitive. Refer to folder structure section (next section) for details about the structure of this folder. | Yes |
65
65
| entryFilePath | Relative path to the root folder of the Spark code/package. The entry file must be either a Python file or a .jar file. | Yes |
66
66
| className | Application's Java/Spark main class | No |
0 commit comments