Fix SparkConnect pod update issue

[Shekharrajak]

Purpose of this PR

Fixes #2784 - SparkConnect server/executor pods not updated when SparkConnect resource spec changes (e.g., image updates).

Proposed changes:

• Add spec hash computation to detect pod spec changes using SHA256

• Implement pod restart logic when spec hash differs from running pod

• Add ServerPodUpdating status condition and Kubernetes events for observability

• Add 22 unit tests for spec change detection and pod restart logic

Change Category

• Bugfix (non-breaking change which fixes an issue)

• Feature (non-breaking change which adds functionality)

• Breaking change (fix or feature that could affect existing functionality)

• Documentation update

Rationale

Kubernetes pods are immutable - spec changes cannot be applied to running pods. The reconciler now computes a hash of the desired pod spec and stores it in a pod annotation. On each reconcile, if the hash differs, the pod is deleted and recreated with the new spec. This follows cloud native patterns (declarative configuration, desired vs actual state comparison).

Checklist

• I have conducted a self-review of my own code.

• I have updated documentation accordingly.

• I have added tests that prove my changes are effective or that my feature works.

• Existing unit tests pass locally with my changes.

Additional Notes

• Hash includes: image, command, args, env vars, volumes, and resources
• Graceful shutdown via existing PreStop lifecycle hook
• Executor pods are handled via ConfigMap updates; new executors use updated template automatically

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yuchaoran2011 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment