Skip to content

Commit ec2c5fa

Browse files
Update apache-hive-warehouse-connector.md
Updated HWC steps for 5.1 clusters
1 parent ef5cee6 commit ec2c5fa

File tree

1 file changed

+28
-5
lines changed

1 file changed

+28
-5
lines changed

articles/hdinsight/interactive-query/apache-hive-warehouse-connector.md

Lines changed: 28 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ ms.topic: how-to
66
author: abhishjain002
77
ms.author: abhishjain
88
ms.reviewer: nijelsf
9-
ms.date: 03/11/2025
9+
ms.date: 08/08/2025
1010
---
1111

1212
# Integrate Apache Spark and Apache Hive with Hive Warehouse Connector in Azure HDInsight
@@ -56,15 +56,13 @@ Hive Warehouse Connector needs separate clusters for Spark and Interactive Query
5656

5757
| HWC Version | Spark Version | InteractiveQuery Version |
5858
|:---:|:---:|---|
59-
| v1 | Spark 2.4 \| HDI 4.0 | Interactive Query 3.1 \| HDI 4.0 |
60-
| v2 | Spark 3.1 \| HDI 5.0 | Interactive Query 3.1 \| HDI 5.0 |
6159
| v2.1 | Spark 3.3.0 \| HDI 5.1 | Interactive Query 3.1 \| HDI 5.1 |
6260

6361
### Create clusters
6462

65-
1. Create an HDInsight Spark **4.0** cluster with a storage account and a custom Azure virtual network. For information on creating a cluster in an Azure virtual network, see [Add HDInsight to an existing virtual network](../../hdinsight/hdinsight-plan-virtual-network-deployment.md#existingvnet).
63+
1. Create an HDInsight Spark **5.1** cluster with a storage account and a custom Azure virtual network. For information on creating a cluster in an Azure virtual network, see [Add HDInsight to an existing virtual network](../../hdinsight/hdinsight-plan-virtual-network-deployment.md#existingvnet).
6664

67-
1. Create an HDInsight Interactive Query (LLAP) **4.0** cluster with the same storage account and Azure virtual network as the Spark cluster.
65+
1. Create an HDInsight Interactive Query (LLAP) **5.1** cluster with the same storage account and Azure virtual network as the Spark cluster.
6866

6967
### Configure HWC settings
7068

@@ -102,6 +100,20 @@ value. The value may be similar to: `thrift://iqgiro.rekufuk2y2cezcbowjkbwfnyvd.
102100

103101
1. Save changes and restart all affected components.
104102

103+
#### Additonal configurations for Spark and Hive
104+
105+
The following configuration needs to be done for **all** head and worker nodes of your Spark and Hive clusters.
106+
107+
1. Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to connect to your Apache Spark and Apache Hive nodes. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
108+
109+
```cmd
110+
111+
```
112+
113+
1. Append the file content of /etc/hosts of hive cluster in /etc/hosts file of spark cluster and vice-versa.
114+
115+
1. Once all nodes are updated then, restart both the clusters.
116+
105117
### Configure HWC for Enterprise Security Package (ESP) clusters
106118
107119
The Enterprise Security Package (ESP) provides enterprise-grade capabilities like Active Directory-based authentication, multi-user support, and role-based access control for Apache Hadoop clusters in Azure HDInsight. For more information on ESP, see [Use Enterprise Security Package in HDInsight](../domain-joined/apache-domain-joined-architecture.md).
@@ -126,8 +138,19 @@ Apart from the configurations mentioned in the previous section, add the followi
126138
127139
* For instance, `hive/hn*.mjry42ikpruuxgs2qy2kpg4q5e.cx.internal.cloudapp.net@PKRSRVUQVMAE6J85.D2.INTERNAL.CLOUDAPP.NET`.
128140
141+
1. The following configuration needs to be done for **all** head and worker nodes of your Spark and Hive clusters.
142+
143+
* Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to connect to your Apache Spark and Apache Hive nodes. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
144+
145+
```cmd
146+
147+
```
148+
149+
* Append tenant domain name (e.g. "abc.contoso.com”) in the last line of /etc/resolv.conf in head and worker nodes of your Spark and Hive clusters.
150+
129151
1. Save changes and restart components as needed.
130152
153+
131154
## Hive Warehouse Connector usage
132155
133156
You can choose between a few different methods to connect to your Interactive Query cluster and execute queries using the Hive Warehouse Connector. Supported methods include the following tools:

0 commit comments

Comments
 (0)