Can I run spark image 3.3.x

[yeldeeb]

I have a dependency to run specific version of Spark 3.3.x, as per documentation only the Stackable Operator for Apache Spark on Kubernetes currently supports the following versions of Spark:
3.5.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
3.4.1 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
3.4.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)

Can I provide my image of spark 3.3.3 and submit it to the operator?

[lfrancke]

We did support 3.3 in our previous release (23.7) and removed it in 23.11.
I do not think we made any breaking changes so I think it'd definitely be worth a shot, yes.

Here's what the Dockerfile looked like at that point: https://github.com/stackabletech/docker-images/blob/23.7.0/spark-k8s/Dockerfile

Please let us know if you need any further help.

[lfrancke]

We have documentation on how to select a custom image.

[yeldeeb]

I used the latest version of stackable spark operator 23.11 then I specified in sparkImage to use 23.7 stackable version and spark version 3.3.0 and it worked!

spec:
  version: "1.0"
  sparkImage:
    stackableVersion: 23.7.0
    productVersion: 3.3.0
  mode: cluster
  mainApplicationFile: local:///stackable/spark-3.3.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.0.jar
  mainClass: org.apache.spark.examples.SparkPi

[razvan]

Spark 3.3 has at least one known bug on K8S that may or may not affect you.

[yeldeeb]

Yeah, It's disappointing.
I have to extend that stackable image to include my custom jars. I guess it will work.