feat: add ingress-agnostic mode for improved compatibility with various ingress controllers

[Denyme24]

Purpose of this PR

fixes #2781

Proposed changes:

• added an opt-in ingress-agnostic mode (Helm value/CLI flag + per-SparkApplication override) to disable nginx-specific path regex mangling and nginx.ingress.kubernetes.io/rewrite-target injection.
• In agnostic mode: keep the user-supplied path as-is, use PathType: Prefix, and do not add nginx rewrite annotations; legacy mode remains the default for backward compatibility.
• Wire the option through controller (web UI + driver ingress), Helm values/deployment, and added tests covering both legacy and agnostic modes; regenerate API types/CRDs as needed.

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[Denyme24]

hey team @ChenYi015 @nabuskey, please review the changes once.
thank you.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from chenyi015. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment