Are there any plans to release spark 3.5.1?

[paulpaul1076]

There are 2 issues in Spark 3.5.0 where there's data loss when performing joins:

https://issues.apache.org/jira/browse/SPARK-45282
https://issues.apache.org/jira/browse/SPARK-45592

So, I was wondering if you're gonna release Spark 3.5.1.

Thanks.

[adwk67]

3.5 is the latest supported version: we are planning for the next release to come out this month, but 3.5.1 is not currently covered.

[supsupsap]

I think it's important to support 3.5.1 because these bugs are critical

[adwk67]

Note (just for the sake of completion): these issues have been fixed in 3.4.2 (supported in our upcoming release).

[sbernauer]

I'm happy to announce that we will ship Spark 3.5.1 in the 24.3 release. The nightly docker image (docker.stackable.tech/stackable/spark-k8s:3.5.1-stackable0.0.0-dev) is already build, the docs and tests are being adopted in stackabletech/spark-k8s-operator#373

[sbernauer]

And docs and tests are merged 👍 Please feel free to give it a try and get back to us in case you notice any problems.

[paulpaul1076]

@sbernauer our devops team will have the time to update this operator next week, that is when I will test it.

[paulpaul1076]

@sbernauer sorry, my company first said that we'll do it next week, them some "big boss" intervened and said that we're not going to use "-dev" versions. It's ridiculous I am sorry that I couldn't help you in testing this.

[sbernauer]

No worries @paulpaul1076! Given it's just a bugfix release I don't expect anything to break. We are in the process of releasing, so you can expect 24.3 next week or the week after ;)

[paulpaul1076]

@sbernauer thank you!