fix(backend): use pipeline version references for pinned recurring runs. Fixes #13933

[demarna1]

Description of your changes

Fixes #13933.

When a recurring run is pinned to a specific pipeline version (the normal UI flow of selecting a version), CreateJob resolved the template at creation time and embedded the compiled Argo workflow into the ScheduledWorkflow CR (spec.workflow.spec) and the stored recurring run (PipelineSpecManifest/WorkflowSpecManifest). This duplicated the durably stored pipeline version and baked in whatever launcher/driver image references were current at creation time — platform upgrades (e.g. patching a CVE in the launcher image) never reached already-created recurring runs.

This PR routes pinned V2 pipeline versions onto the reference-based path that already exists for the "always use latest" case (#13440):

• CreateJob now stores only pipelineId/pipelineVersionId on the ScheduledWorkflow (plus runtime parameters and pipeline root) and persists the recurring run without a manifest. Job inputs are still validated up front via ValidateJobInputs.
• At trigger time, the ScheduledWorkflow controller's existing reference branch calls RunService.CreateRun with a PipelineVersionReference carrying the pinned PipelineVersionId, so the run is compiled fresh from the stored pipeline version with the platform's current images. No controller changes were needed — this branch already forwards a non-empty pinned version id.
• The duplicated reference-SWF construction is extracted into a referenceScheduledWorkflow helper shared with the "latest" branch, and the manifest-persistence gate is now an explicit embedManifest flag instead of relying on tmpl nil-ness.

Behavior preserved:

• Raw manifest submissions (PipelineSpecManifest/WorkflowSpecManifest supplied) keep embedding, as before.
• V1 (Argo) templates keep embedding, including the v1beta1 API's pin-latest-at-creation shim.
• Existing recurring runs are unaffected: they keep their persisted manifests and continue to be reconciled by ReconcileSwfCrs; new pinned jobs are skipped by its existing empty-manifest check, same as latest-path jobs.
• Like the latest path, the recurring run's service account is no longer defaulted from the compiled spec at creation time; the default is applied when the run is compiled at trigger time.

Per-trigger compilation cost matches what the "latest" path already incurs today.

Testing

• New TestCreateJob_ThroughPipelineVersionV2: pinned V2 version → stored job has no manifest, SWF carries pipelineVersionId with spec.workflow.spec absent and runtime parameters present.
• New TestCreateJob_ThroughPipelineVersionV2_InvalidParams: extra runtime parameter is rejected at creation time.
• New TestSubmitNewWorkflowIfNotAlreadySubmitted_PipelineVersionReference: closes an existing coverage gap on the controller's reference branch — asserts no Argo workflow is created directly and the CreateRunRequest carries the pinned PipelineVersionReference, recurring run id, and translated runtime config.
• Existing pinned-V1 and raw-manifest tests (TestCreateJob_ThroughPipelineVersion, TestCreateJob_ThroughPipelineIdAndPipelineVersion, TestCreateJob_ThroughWorkflowSpec*, TestReconcileSwfCrs, v1 job_server tests) pass unmodified.
• Full backend unit sweep (per repo guide, excluding integration/compiler/E2E suites) passes.

Checklist

• The title of your PR is conformant to the contribution guidelines

🤖 Generated with Claude Code

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nsingla for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• backend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

Hi @demarna1. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.