[SPARK-54428] Upgrade Spark to 4.1.0-preview4

### What changes were proposed in this pull request?

This PR aims to upgrade Spark to `4.1.0-preview4`.

### Why are the changes needed?

To use the latest bug fixed versions and be ready for upcoming Apache Spark 4.1.0.

### Does this PR introduce _any_ user-facing change?

No behavior change.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #418 from dongjoon-hyun/SPARK-54428.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

gradle/libs.versions.toml

@@ -19,7 +19,7 @@ fabric8 = "7.4.0"
 lombok = "1.18.42"
 operator-sdk = "5.1.4"
 dropwizard-metrics = "4.2.33"
-spark = "4.1.0-preview3"
+spark = "4.1.0-preview4"
 log4j = "2.24.3"
 slf4j = "2.0.17"