[SPARK-52250] Promote Spark CRDs to v1beta1

[dongjoon-hyun]

What changes were proposed in this pull request?

In line with Apache Spark 4.0.0 release, this PR aims to promote Spark CRDs from v1alpha1 to v1beta1 at Apache Spark K8s Operator 0.3.0.

Why are the changes needed?

To show the maturity of these CRDs according to K8s API versioning.

• https://kubernetes.io/docs/reference/using-api/

Version	Description
Alpha	The software is recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support
Beta	The software is not recommended for production uses. The support for a feature will not be dropped, though the details may change.
Stable	The version name is vX where X is an integer

Does this PR introduce any user-facing change?

Yes, the existing CRDs should be removed cleanly before installing v1beta1. However, we are still in 0.3.0.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @viirya , @peter-toth , @jiangzho

This is the first step of Apache Spark K8s Operator CRD graduation (Stable) in the future and to protect the existing features (the definition of Beta). I'm working on several items like the followings for 0.3.0 milestone.

• Auto-upgrading from v1alpha1 to v1beta1 in an independent activity.
• Upgrade CRD generating tool

[dongjoon-hyun]

Thank you, @viirya . Merged to main.