Skip to content

Conversation

razvan
Copy link
Member

@razvan razvan commented May 27, 2025

Description

Since Spark 3.5.5 the tests fails because of the error below.

I think it's safe to remove this test.

	25/05/26 15:28:43 INFO S3AStoreImpl: Interrupted: aborting upload
	Exception in thread "main" org.apache.spark.SparkException: Uploading file /stackable/spark/examples/jars/spark-examples_2.12-3.5.5.jar failed...
	    at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:327)
	    at org.apache.spark.deploy.k8s.KubernetesUtils$.$anonfun$uploadAndTransformFileUris$1(KubernetesUtils.scala:276)
	    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	    at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadAndTransformFileUris(KubernetesUtils.scala:275)
	    at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.$anonfun$getAdditionalPodSystemProperties$1(BasicDriverFeatureStep.scala:180)
	    at scala.collection.immutable.List.foreach(List.scala:431)
	    at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.getAdditionalPodSystemProperties(BasicDriverFeatureStep.scala:170)
	    at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$5(KubernetesDriverBuilder.scala:86)
	    at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	    at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	    at scala.collection.immutable.List.foldLeft(List.scala:91)
	    at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:84)
	    at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:104)
	    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6(KubernetesClientApplication.scala:256)
	    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$6$adapted(KubernetesClientApplication.scala:250)
	    at org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:48)
	    at org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:46)
	    at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:94)
	    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:250)
	    at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:223)
	    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
	    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
	    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
	    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
	    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
	    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	Caused by: org.apache.spark.SparkException: Error uploading file spark-examples_2.12-3.5.5.jar
	    at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileToHadoopCompatibleFS(KubernetesUtils.scala:350)
	    at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:323)
	    ... 32 more
	Caused by: org.apache.hadoop.fs.s3a.AWSBadRequestException: upload on spark-upload-587deb01-2c17-4297-a409-bcb28606ef9c/spark-examples_2.12-3.5.5.jar: software.amazon.awssdk.services.s3.mo
	del.S3Exception: Missing fields in request. (Service: S3, Status Code: 400, Request ID: 18431D5316A1780B, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e
	3e8):MissingFields: Missing fields in request. (Service: S3, Status Code: 400, Request ID: 18431D5316A1780B, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7a
	c2e3e8)
	    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:261)

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Roadmap has been updated

@razvan razvan self-assigned this May 27, 2025
@razvan razvan moved this to Development: Waiting for Review in Stackable Engineering May 27, 2025
@razvan razvan requested a review from a team May 27, 2025 08:11
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is what it is I guess

@sbernauer sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering May 27, 2025
@razvan razvan added this pull request to the merge queue May 27, 2025
Merged via the queue into main with commit 8f42276 May 27, 2025
17 checks passed
@razvan razvan deleted the fix/remove-public-s3-test branch May 27, 2025 08:50
@sbernauer sbernauer moved this from Development: In Review to Development: Done in Stackable Engineering Jun 2, 2025
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants