[SPARK-53911] Support SPARK_VERSION placeholder in container image names

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR allows users to use {{SPARK_VERSION}} in the following three configs.

spark.kubernetes.container.image
spark.kubernetes.driver.container.image
spark.kubernetes.executor.container.image

Why are the changes needed?

We should implement this in Apache Spark K8s Operator for feature parity with Apache Spark 4.1.0. Otherwise, it will be replaced with the compile dependency, 4.1.0-preview2 instead of the value of RuntimeVersions.sparkVersion.

• [SPARK-52902][K8S] Support SPARK_VERSION placeholder in container image names spark#51592

Does this PR introduce any user-facing change?

No because {{SPARK_VERSION}} is a newly supported pattern.

How was this patch tested?

Pass the CIs with newly added test cases.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @peter-toth

[viirya]

Do we need to add some examples for this?

[dongjoon-hyun]

Thank you. We can add two examples for both SparkApplication and SparkCluster. So, I didn't add them here.

[dongjoon-hyun]

Merged to main.

[peter-toth]

Late LGTM.