You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/azure-cli-samples.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,9 +22,9 @@ This article provides sample scripts for common tasks. For each example, update
22
22
23
23
* Optional: Bash. The examples in this article use the Bash shell on Windows 10. See [Windows Subsystem for Linux Installation Guide for Windows 10](/windows/wsl/install-win10) for installation steps. The examples work from a Windows Command prompt with some slight modifications.
24
24
25
-
## az login
25
+
## `az login`
26
26
27
-
[Log in to Azure](/cli/azure/reference-index#az-login).
27
+
[Sign in to Azure](/cli/azure/reference-index#az-login).
28
28
29
29
```azurecli
30
30
az login
@@ -33,7 +33,7 @@ az login
33
33
# az account set --subscription "SUBSCRIPTIONID"
34
34
```
35
35
36
-
## az hdinsight create
36
+
## `az hdinsight create`
37
37
38
38
[Creates a new cluster](/cli/azure/hdinsight#az-hdinsight-create).
39
39
@@ -166,7 +166,7 @@ az hdinsight create \
166
166
--cluster-configuration $clusterConfiguration
167
167
```
168
168
169
-
## az hdinsight application create
169
+
## `az hdinsight application create`
170
170
171
171
[Create an application for a HDInsight cluster](/cli/azure/hdinsight/application#az-hdinsight-application-create).
172
172
@@ -233,7 +233,7 @@ az hdinsight application create \
233
233
--sub-domain-suffix $subDomainSuffix
234
234
```
235
235
236
-
## az hdinsight script-action execute
236
+
## `az hdinsight script-action execute`
237
237
238
238
[Execute script actions on the specified HDInsight cluster](/cli/azure/hdinsight/script-action#az-hdinsight-script-action-execute).
Copy file name to clipboardExpand all lines: articles/hdinsight/domain-joined/apache-domain-joined-configure-using-azure-adds.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,9 +9,9 @@ ms.date: 09/06/2024
9
9
10
10
# Configure HDInsight clusters for Microsoft Entra integration with Enterprise Security Package
11
11
12
-
This article provides a summary and overview of the process of creating and configuring an HDInsight cluster integrated with Microsoft Entra ID. This integration relies on a HDInsight feature called Enterprise Security Package (ESP), Microsoft Entra Domain Services and your pre-existing on-premises Active Directory.
12
+
This article provides a summary and overview of the process of creating and configuring an HDInsight cluster integrated with Microsoft Entra ID. This integration relies on a HDInsight feature called Enterprise Security Package (ESP), Microsoft Entra Domain Services and your preexisting on-premises Active Directory.
13
13
14
-
For a detailed, step-by-step tutorial on setting up and configuring a domain in Azure and creating an ESP enabled cluster and then syncing on-premises users, see [Create and configure Enterprise Security Package clusters in Azure HDInsight](apache-domain-joined-create-configure-enterprise-security-cluster.md).
14
+
For a detailed, step-by-step tutorial on setting up and configuring a domain in Azure and creating an ESP enabled cluster and then syncing on-premises users, see [Create, and configure Enterprise Security Package clusters in Azure HDInsight](apache-domain-joined-create-configure-enterprise-security-cluster.md).
15
15
16
16
## Background
17
17
@@ -30,7 +30,7 @@ There are a few prerequisites to complete before you can create an ESP-enabled H
30
30
- Create and authorize a managed identity.
31
31
- Complete Networking setup for DNS and related issues.
32
32
33
-
Each of these items are discussed in details. For a walkthrough of completing all of these steps, see [Create and configure Enterprise Security Package clusters in Azure HDInsight](apache-domain-joined-create-configure-enterprise-security-cluster.md).
33
+
Each of this item is discussed in details. For a walkthrough of completing all of these steps, see [Create and configure Enterprise Security Package clusters in Azure HDInsight](apache-domain-joined-create-configure-enterprise-security-cluster.md).
### Check Microsoft Entra Domain Services health status
68
68
69
-
View the health status of Microsoft Entra Domain Services by selecting **Health** in the **Manage** category. Make sure the status of Microsoft Entra Domain Services is green (running) and the synchronization is complete.
69
+
View the health status of Microsoft Entra Domain Services by selecting `Health` in the `Manage` category. Make sure the status of Microsoft Entra Domain Services is green (running) and the synchronization is complete.
@@ -95,7 +95,7 @@ For example, the Microsoft Entra Domain Services admin can assign this role to t
95
95
> [!NOTE]
96
96
> Microsoft Entra Domain Services must be deployed in an Azure Resource Manager-based virtual network. Classic virtual networks are not supported for Microsoft Entra Domain Services. For more information, see [Enable Microsoft Entra Domain Services by using the Azure portal](../../active-directory-domain-services/tutorial-create-instance-advanced.md#create-and-configure-the-virtual-network).
97
97
98
-
Enable Microsoft Entra Domain Services. Then a local Domain Name System (DNS) server runs on the Active Directory virtual machines (VMs). Configure your Microsoft Entra Domain Services virtual network to use these custom DNS servers. To locate the right IP addresses, select **Properties** in the **Manage** category and look under **IP ADDRESS ON VIRTUAL NETWORK**.
98
+
Enable Microsoft Entra Domain Services. Then a local Domain Name System (DNS) server runs on the Active Directory virtual machines (VMs). Configure your Microsoft Entra Domain Services virtual network to use these custom DNS servers. To locate the right IP addresses, select `Properties` in the `Manage` category and look under **IP ADDRESS ON VIRTUAL NETWORK**.
99
99
100
100
:::image type="content" source="./media/apache-domain-joined-configure-using-azure-adds/hdinsight-aadds-dns1.png" alt-text="Locate IP addresses for local DNS servers." border="true":::
101
101
@@ -105,7 +105,7 @@ Change the configuration of the DNS servers in the Microsoft Entra Domain Servic
105
105
106
106
It's easier to place both the Microsoft Entra Domain Services instance and the HDInsight cluster in the same Azure virtual network. If you plan to use different virtual networks, you must peer those virtual networks so that the domain controller is visible to HDInsight VMs. For more information, see [Virtual network peering](../../virtual-network/virtual-network-peering-overview.md).
107
107
108
-
After the virtual networks are peered, configure the HDInsight virtual network to use a custom DNS server. And enter the Microsoft Entra Domain Services private IPs as the DNS server addresses. When both virtual networks use the same DNS servers, your custom domain name resolves to the right IP and it is reachable from HDInsight. For example, if your domain name is `contoso.com`, then after this step, `ping contoso.com` should resolve to the right Microsoft Entra Domain Services IP.
108
+
After the virtual networks are peered, configure the HDInsight virtual network to use a custom DNS server. And enter the Microsoft Entra Domain Services private IPs as the DNS server addresses. When both virtual networks use the same DNS servers, your custom domain name resolves to the right IP and it's reachable from HDInsight. For example, if your domain name is `contoso.com`, then after this step, `ping contoso.com` should resolve to the right Microsoft Entra Domain Services IP.
109
109
110
110
:::image type="content" source="./media/apache-domain-joined-configure-using-azure-adds/hdinsight-aadds-peered-vnet-configuration.png" alt-text="Configuring custom DNS servers for a peered virtual network." border="true":::
In the **Upload New File** dialog box, under **File name**, select **Browse**. In the **Upload Blob** dialog box, go to the *bin\debug* folder for the *mapper* project, and then choose the *mapper.exe* file. Finally, select **Open** and then **OK** to complete the upload.
160
+
In the **Upload New File** dialog box, under **File name**, select **Browse**. In the **Upload Blob** dialog box, go to the `bin\debug` folder for the *mapper* project, and then choose the *mapper.exe* file. Finally, select **Open** and then **OK** to complete the upload.
161
161
162
162
* For **Azure Data Lake Storage**, right-click an empty area in the file listing, and then select **Upload**. Finally, select the *mapper.exe* file and then select **Open**.
163
163
@@ -243,7 +243,7 @@ The following procedure describes how to run a MapReduce job using an SSH sessio
243
243
244
244
Use the following PowerShell script to run a MapReduce job and download the results.
This script prompts you for the cluster login account name and password, along with the HDInsight cluster name. Once the job completes, the output is downloaded to a file named *output.txt*. The following text is an example of the data in the `output.txt` file:
Copy file name to clipboardExpand all lines: articles/hdinsight/hadoop/apache-hadoop-use-hive-curl.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -178,7 +178,7 @@ $clusterName
178
178
179
179
1. Once the state of the job has changed to **SUCCEEDED**, you can retrieve the results of the job from Azure Blob storage. The `statusdir` parameter passed with the query contains the location of the output file;in this case, `/example/rest`. This address stores the output in the `example/curl` directory in the clusters default storage.
180
180
181
-
You can list and download these files by using the [Azure CLI](/cli/azure/install-azure-cli). For more information on using the Azure CLI with Azure Storage, see the [Use Azure CLI with Azure Storage](../../storage/blobs/storage-quickstart-blobs-cli.md) document.
181
+
You can list and download these files by using the [Azure CLI](/cli/azure/install-azure-cli). For more information, see [Use Azure CLI with Azure Storage](../../storage/blobs/storage-quickstart-blobs-cli.md) document.
Copy file name to clipboardExpand all lines: articles/hdinsight/hadoop/hdinsight-use-sqoop.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,14 +48,14 @@ HDInsight cluster comes with some sample data. You use the following two samples
48
48
| state |string |
49
49
| country |string |
50
50
| querydwelltime |double |
51
-
| sessionid |bigint |
51
+
|`sessionid`|bigint |
52
52
| sessionpagevieworder |bigint |
53
53
54
54
In this article, you use these two datasets to test Sqoop import and export.
55
55
56
56
## <aname="create-cluster-and-sql-database"></a>Set up test environment
57
57
58
-
The cluster, SQL database, and other objects are created through the Azure portal using an Azure Resource Manager template. The template can be found in [Azure quickstart templates](https://azure.microsoft.com/resources/templates/hdinsight-linux-with-sql-database/). The Resource Manager template calls a bacpac package to deploy the table schemas to an SQL database. If you want to use a private container for the bacpac files, use the following values in the template:
58
+
The cluster, SQL database, and other objects are created through the Azure portal using an Azure Resource Manager template. The template can be found in [Azure Quickstart Templates](https://azure.microsoft.com/resources/templates/hdinsight-linux-with-sql-database/). The Resource Manager template calls a bacpac package to deploy the table schemas to an SQL database. If you want to use a private container for the bacpac files, use the following values in the template:
59
59
60
60
```json
61
61
"storageKeyType": "Primary",
@@ -77,11 +77,11 @@ The cluster, SQL database, and other objects are created through the Azure porta
77
77
|Resource group |Select your resource group from the drop-down list, or create a new one|
78
78
|Location |Select a region from the drop-down list.|
79
79
|Cluster Name |Enter a name for the Hadoop cluster. Use lowercase letter only.|
80
-
|Cluster Login User Name |Keep the pre-populated value `admin`.|
81
-
|Cluster Login Password |Enter a password.|
82
-
|Ssh User Name |Keep the pre-populated value `sshuser`.|
80
+
|Cluster sign-in User Name |Keep the prepopulated value `admin`.|
81
+
|Cluster sign in Password |Enter a password.|
82
+
|Ssh User Name |Keep the prepopulated value `sshuser`.|
83
83
|Ssh Password |Enter a password.|
84
-
|Sql Admin Login|Keep the pre-populated value `sqluser`.|
84
+
|Sql Admin sign-in|Keep the prepopulated value `sqluser`.|
85
85
|Sql Admin Password |Enter a password.|
86
86
|_artifacts Location | Use the default value unless you want to use your own bacpac file in a different location.|
87
87
|_artifacts Location Sas Token |Leave blank.|
@@ -100,7 +100,7 @@ HDInsight can run Sqoop jobs by using various methods. Use the following table t
100
100
101
101
|**Use this** if you want... | ...an **interactive** shell | ...**batch** processing | ...from this **client operating system**|
102
102
|:--- |:---:|:---:|:--- |:--- |
103
-
|[SSH](apache-hadoop-use-sqoop-mac-linux.md)|? |? |Linux, Unix, Mac OS X, or Windows |
103
+
|[SSH](apache-hadoop-use-sqoop-mac-linux.md)|? |? |Linux, Unix, macOS X, or Windows |
104
104
|[.NET SDK for Hadoop](apache-hadoop-use-sqoop-dotnet-sdk.md)| |? |Windows (for now) |
Copy file name to clipboardExpand all lines: articles/hdinsight/hadoop/python-udf-hdinsight.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -99,7 +99,7 @@ The script output is a concatenation of the input values for `devicemake` and `d
99
99
100
100
### Upload file (shell)
101
101
102
-
The following command, replaces `sshuser` with the actual username if different. Replace `mycluster` with the actual cluster name. Ensure your working directory is where the file is located.
102
+
The following command replaces `sshuser` with the actual username if different. Replace `mycluster` with the actual cluster name. Ensure your working directory is where the file is located.
103
103
104
104
1. Use `scp` to copy the files to your HDInsight cluster. Edit and enter the command:
105
105
@@ -393,7 +393,7 @@ In the commands below, replace `sshuser` with the actual username if different.
393
393
DUMP DETAILS;
394
394
```
395
395
396
-
3. After entering the following line, the job should start. Once the job completes, it returns output similar to the following data:
396
+
3. After you enter the following line, the job should start. Once the job completes, it returns output similar to the following data:
397
397
398
398
```output
399
399
((2012-02-03,20:11:56,SampleClass5,[TRACE],verbose detail for id 990982084))
@@ -569,13 +569,13 @@ This problem may be caused by the line endings in the Python file. Many Windows
569
569
570
570
You can use the following PowerShell statements to remove the CR characters before uploading the file to HDInsight:
Both of the example PowerShell scripts used to run the examples contain a commented line that displays error output for the job. If you aren't seeing the expected output for the job, uncomment the following line and see if the error information indicates a problem.
This error points to a problem with custom DNS configuration. DNS servers within a virtual network can forward DNS queries to Azure's recursive resolvers to resolve hostnames within that virtual network (see [Name Resolution in Virtual Networks](../../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md) for details). Access to Azure's recursive resolvers is provided via the virtual IP 168.63.129.16. This IP is only accessible from the Azure VMs. It is nonfunctional if you are using an OnPrem DNS server, or your DNS server is an Azure VM, which is not part of the cluster's virtual network.
23
+
This error points to a problem with custom DNS configuration. DNS servers within a virtual network can forward DNS queries to Azure's recursive resolvers to resolve hostnames within that virtual network (see [Name Resolution in Virtual Networks](../../virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances.md) for details). Access to Azure's recursive resolvers is provided via the virtual IP 168.63.129.16. This IP is only accessible from the Azure VMs. It's nonfunctional if you're using an on-premises DNS server, or your DNS server is an Azure VM, which isn't part of the cluster's virtual network.
24
24
25
25
### Resolution
26
26
@@ -50,11 +50,11 @@ Azure Storage and SQL don't have fixed IP Addresses, so we need to allow outboun
50
50
51
51
### Resolution
52
52
53
-
* If your cluster uses a [Network Security Group (NSG)](../../virtual-network/virtual-network-vnet-plan-design-arm.md).
53
+
* If your cluster uses a [Network Security Group (NSG)](../../virtual-network/virtual-network-vnet-plan-design-arm.md)
54
54
55
55
Go to the Azure portal and identify the NSG that is associated with the subnet where the cluster is being deployed. In the **Outbound security rules** section, allow outbound access to internet without limitation (note that a smaller **priority** number here means higher priority). Also, in the **subnets** section, confirm if this NSG is applied to the cluster subnet.
56
56
57
-
* If your cluster uses a [User-defined Routes (UDR)](../../virtual-network/virtual-networks-udr-overview.md).
57
+
* If your cluster uses a [User-defined Routes (UDR)](../../virtual-network/virtual-networks-udr-overview.md)
58
58
59
59
Go to the Azure portal and identify the route table that is associated with the subnet where the cluster is being deployed. Once you find the route table for the subnet, inspect the **routes** section in it.
60
60
@@ -68,14 +68,14 @@ Error description contains "Failed to establish an outbound connection from the
68
68
69
69
### Cause
70
70
71
-
When using Private Linked HDInsight clusters, outbound access from the cluster must be configured to allow connections to be made to the HDInsight resource provider.
71
+
When you use Private Linked HDInsight clusters, outbound access from the cluster must be configured to allow connections to be made to the HDInsight resource provider.
72
72
73
73
### Resolution
74
74
75
75
* To resolve this issue, refer to the HDInsight Private Link setup steps at [private link setup](../hdinsight-private-link.md)
76
76
---
77
77
78
-
## "Virtual network configuration is not compatible with HDInsight requirement"
78
+
## "Virtual network configuration isn't compatible with HDInsight requirement"
79
79
80
80
### Issue
81
81
@@ -116,7 +116,7 @@ Validate that 168.63.129.16 is in the custom DNS chain. DNS servers within a vir
116
116
117
117
Based on the result - choose one of the following steps to follow:
118
118
119
-
#### 168.63.129.16 is not in this list
119
+
#### 168.63.129.16 isn't in this list
120
120
121
121
**Option 1**
122
122
Add 168.63.129.16 as the first custom DNS for the virtual network using the steps described in [Plan a virtual network for Azure HDInsight](../hdinsight-plan-virtual-network-deployment.md). These steps are applicable only if your custom DNS server runs on Linux.
@@ -145,7 +145,7 @@ Another cause for this `InvalidNetworkConfigurationErrorCode` error code could b
145
145
146
146
### Resolution
147
147
148
-
Use the valid parameters for `Get-AzVirtualNetwork` as documented in the [Az PowerShell SDK](/powershell/module/az.network/get-azvirtualnetwork)
148
+
Use the valid parameters for `Get-AzVirtualNetwork` as documented in the [Az PowerShell module PowerShell SDK](/powershell/module/az.network/get-azvirtualnetwork)
0 commit comments