[SPARK-50670] Use Spark 3.5.4 in E2E tests

### What changes were proposed in this pull request?

This PR aims to use Spark 3.5.4 in E2E tests.

### Why are the changes needed?

To use the latest Spark versions in testing.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #149 from dongjoon-hyun/SPARK-50670.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

tests/e2e/python/chainsaw-test.yaml

@@ -23,11 +23,11 @@ spec:
   scenarios:
     - bindings:
         - name: "SPARK_VERSION"
-          value: "3.5.3"
+          value: "3.5.4"
         - name: "SCALA_VERSION"
           value: "2.12"
         - name: "IMAGE"
-          value: "apache/spark:3.5.3-scala2.12-java17-python3-ubuntu"
+          value: "apache/spark:3.5.4-scala2.12-java17-python3-ubuntu"
   steps:
     - name: install-spark-application
       try:

tests/e2e/spark-versions/chainsaw-test.yaml

@@ -32,13 +32,13 @@ spec:
         value: "apache/spark:4.0.0-preview2-scala2.13-java17-ubuntu"
   - bindings:
       - name: "SPARK_VERSION"
-        value: "3.5.3"
+        value: "3.5.4"
       - name: "SCALA_VERSION"
         value: "2.12"
       - name: "JAVA_VERSION"
         value: "17"
       - name: "IMAGE"
-        value: 'apache/spark:3.5.3-scala2.12-java17-ubuntu'
+        value: 'apache/spark:3.5.4-scala2.12-java17-ubuntu'
   - bindings:
       - name: "SPARK_VERSION"
         value: "3.4.3"