Merge pull request #269 from marklogic/feature/bump-spark

Testing against Spark 3.4.3

Author: rjrudin

CONTRIBUTING.md

@@ -176,8 +176,8 @@ spark.read.option("header", True).csv("src/test/resources/data.csv")\
 When you run PySpark, it will create its own Spark cluster. If you'd like to try against a separate Spark cluster
 that still runs on your local machine, perform the following steps:
 
-1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.1` since we are currently
-building against Spark 3.4.1.
+1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.3` since we are currently
+building against Spark 3.4.3.
 2. `cd ~/.sdkman/candidates/spark/current/sbin`, which is where sdkman will install Spark.
 3. Run `./start-master.sh` to start a master Spark node.
 4. `cd ../logs` and open the master log file that was created to find the address for the master node. It will be in a

examples/entity-aggregation/build.gradle

@@ -7,7 +7,7 @@ repositories {
 }
 
 dependencies {
-  implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
+  implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
   implementation "com.marklogic:marklogic-spark-connector:2.2.0"
   implementation "org.postgresql:postgresql:42.6.2"
 }

examples/java-dependency/build.gradle

@@ -7,7 +7,7 @@ repositories {
 }
 
 dependencies {
-  implementation 'org.apache.spark:spark-sql_2.12:3.4.1'
+  implementation 'org.apache.spark:spark-sql_2.12:3.4.3'
   implementation 'com.marklogic:marklogic-spark-connector:2.2.0'
 }
 

gradle.properties

@@ -1,13 +1,6 @@
-# Testing against 3.3.2 for the 2.0.0 release as 3.3.0 was released in June 2022 and 3.3.2 in February 2023, while
-# 3.4.0 is fairly new - April 2023. And at least AWS Glue and EMR are only on 3.3.0. But 3.3.2 has bug fixes that
-# affect some of our tests - see PushDownGroupByCountTest for an example. So we're choosing to build and test
-# against the latest 3.3.x release so we're not writing assertions based on buggy behavior in Spark 3.3.0.
-#
-# For 2.1.0, planning on using at least 3.4.x, and possibly 3.5.x. All tests are passing with 3.4.x when authors are
-# in a single document on MarkLogic 11. The tests that verify the number of rows read from MarkLogic (as opposed to
-# rows in the Spark dataset) will fail on MarkLogic 12 for now given that all rows come from the same document, and thus
-# all come from a single call to MarkLogic.
-sparkVersion=3.4.1
+# Staying with 3.4.x for now, as some pushdown tests are failing when using 3.5.x.
+# 3.4.3 release notes - https://spark.apache.org/releases/spark-release-3-4-3.html .
+sparkVersion=3.4.3
 
 # Only used for the test app and for running tests.
 mlHost=localhost