Skip to content

Commit 68aeaef

Browse files
authored
Update hdinsight-50-component-versioning.md
1 parent ade96b3 commit 68aeaef

File tree

1 file changed

+24
-0
lines changed

1 file changed

+24
-0
lines changed

articles/hdinsight/hdinsight-50-component-versioning.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,30 @@ This table lists certain HDInsight 4.0 cluster types that have retired or will b
4141
| HDInsight 4.0 Kafka | 1.1 | Dec 31, 2020 | Dec 31, 2020 |
4242
| HDInsight 4.0 Kafka | 2.1.0 | Sep 30, 2022 | Oct 1, 2022 |
4343

44+
## Spark
45+
46+
:::image type="content" source="./media/hdinsight-release-notes/spark-3-1-for-hdi-5-0.png" alt-text="Screenshot of Spark 3.1 for HDI 5.0":::
47+
48+
> [!NOTE]
49+
> * If you are using Azure User Interface to create a Spark Cluster for HDInsight, you will see from the dropdown list an additional version Spark 3.1.(HDI 5.0) along with the older versions. This version is a renamed version of Spark 3.1.(HDI 4.0) and it is backward compatible.
50+
> * This is only an UI level change, which doesn’t impact anything for the existing users and users who are already using the ARM template to build their clusters. For backward compatibility, ARM supports creating spark 4.0 and 5.0 for 3.1 version which maps to same versions spark 3.1 (5.0)
51+
52+
## Interactive Query
53+
54+
:::image type="content" source="./media/hdinsight-release-notes/interactive-query-3-1-for-hdi-5-0.png" alt-text="Screenshot of interactive query 3.1 for HDI 5.0":::
55+
56+
> [!NOTE]
57+
> * If you are creating an Interactive Query Cluster, you will see from the dropdown list an other version as Interactive Query 3.1 (HDI 5.0).
58+
> * If you are going to use Spark 3.1 version along with Hive which require ACID support, you need to select this version Interactive Query 3.1 (HDI 5.0).
59+
60+
## Kafka
61+
62+
**Known Issue** – Current ARM template supports only 4.0 even though it shows 5.0 image in portal Cluster creation may fail with the following error message if you select version 5.0 in the UI.
63+
64+
`HDI Version'5.0" is not supported for clusterType ''Kafka" and component Version ‘2.4'.,Cluster component version is not applicable for HDI version: 5.0 cluster type: KAFKA (Code: BadRequest)`
65+
66+
We are working on this issue, and a fix will be rolled out shortly.
67+
4468

4569
## Next steps
4670

0 commit comments

Comments
 (0)