Skip to content

Commit 1d3dd58

Browse files
committed
[SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zookeeper 3.4.7 test dependency to Kafka modules
## What changes were proposed in this pull request? This is a followup of apache#23099 . After upgrading to Kafka 2.1.0, maven test fails due to Zookeeper test dependency while sbt test succeeds. - [sbt test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/5203/) - [maven test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5653/) The root cause is that the embedded Kafka server is using [Zookeepr 3.4.7 API](https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html ) while Apache Spark provides Zookeeper 3.4.6. This PR adds a test dependency. ``` KafkaMicroBatchV2SourceSuite: *** RUN ABORTED *** ... org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123) ... Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) at kafka.server.KafkaServer.startup(KafkaServer.scala:202) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116) ... ``` ## How was this patch tested? Pass the maven Jenkins test. Closes apache#23119 from dongjoon-hyun/SPARK-25954-2. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 0ec7b99 commit 1d3dd58

File tree

2 files changed

+14
-0
lines changed

2 files changed

+14
-0
lines changed

external/kafka-0-10-sql/pom.xml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,13 @@
8989
</exclusion>
9090
</exclusions>
9191
</dependency>
92+
<!-- Kafka embedded server uses Zookeeper 3.4.7 API -->
93+
<dependency>
94+
<groupId>org.apache.zookeeper</groupId>
95+
<artifactId>zookeeper</artifactId>
96+
<version>3.4.7</version>
97+
<scope>test</scope>
98+
</dependency>
9299
<dependency>
93100
<groupId>net.sf.jopt-simple</groupId>
94101
<artifactId>jopt-simple</artifactId>

external/kafka-0-10/pom.xml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,13 @@
7474
</exclusion>
7575
</exclusions>
7676
</dependency>
77+
<!-- Kafka embedded server uses Zookeeper 3.4.7 API -->
78+
<dependency>
79+
<groupId>org.apache.zookeeper</groupId>
80+
<artifactId>zookeeper</artifactId>
81+
<version>3.4.7</version>
82+
<scope>test</scope>
83+
</dependency>
7784
<dependency>
7885
<groupId>net.sf.jopt-simple</groupId>
7986
<artifactId>jopt-simple</artifactId>

0 commit comments

Comments
 (0)