Stop embedding compiled workflow manifests in ScheduledWorkflow for version-pinned recurring runs #13933

Summary

When a recurring run is created without pinning a pipeline version ("always use the latest version"), the ScheduledWorkflow controller creates runs by calling back into the API server's RunService.CreateRun with just PipelineId/PipelineVersionId, letting the API server resolve and compile the template fresh at trigger time. This avoids storing a compiled Argo manifest in the CRD.

However, when a recurring run is pinned to a specific pipeline version — including via the normal UI flow of selecting a version when creating a recurring run — the compiled workflow manifest is resolved once at creation time and baked into the Job/ScheduledWorkflow spec (WorkflowSpecManifest/PipelineSpecManifest). This is the same code path used for legacy, fully-freeform manifest submission.

This means the "reference at trigger time" architecture that already exists for the "always latest" case isn't actually available for the much more common production pattern of pinning to a specific, validated pipeline version.

Problems this causes

1. Duplication of state. The compiled manifest stored in the ScheduledWorkflow CRD duplicates the pipeline version's spec, which is already stored durably (pipeline/pipeline-version tables + object store). The CRD becomes a second, driftable copy of the same logical artifact.
2. Image lock-in / vulnerability exposure. Because the manifest is compiled once at recurring-run-creation time, it bakes in whatever KFP launcher/driver/sidecar image references were current then. Upgrading the KFP platform (e.g. patching a CVE in the launcher image) has no effect on already-created recurring runs — they keep running the old images indefinitely unless the ScheduledWorkflow/recurring run is deleted and recreated. In practice this means fleets of recurring runs silently drift out of sync with the platform's current image versions and accumulate unpatched vulnerabilities.

Where this happens (master branch)

• backend/src/apiserver/resource/resource_manager.go, recurring run creation path:

  // If the pipeline version or pipeline spec is provided, this means the user wants to pin to a specific pipeline.
  // Otherwise, always let the ScheduledWorkflow controller pick the latest.
  if job.PipelineVersionId != "" || job.PipelineSpecManifest != "" || job.WorkflowSpecManifest != "" {
      // resolves template now and stores the compiled manifest on the Job / ScheduledWorkflow
  }

  A non-empty PipelineVersionId — even one referencing a normal, stored pipeline version rather than a raw uploaded manifest — takes the same "resolve and embed now" branch as legacy freeform manifest submission.

• backend/src/apiserver/resource/resource_manager.go, ReconcileSwfCrs:

  if jobs[i].PipelineSpec.PipelineSpecManifest == "" && jobs[i].PipelineSpec.WorkflowSpecManifest == "" {
      continue // not pinned via manifest -> reference-based path used instead
  }

  Only jobs with both manifest fields empty skip embedding and rely on the reference-based path.

• backend/src/crd/controller/scheduledworkflow/controller.go, submitNewWorkflowIfNotAlreadySubmitted: already supports calling c.runClient.CreateRun() with a PipelineVersionReference{PipelineId, PipelineVersionId} where PipelineVersionId "can be empty, which causes the latest pipeline version to be selected" — this is the exact mechanism that should also be used for a non-empty, pinned PipelineVersionId.

Reproduction

1. Deploy KFP 2.x with a V2 pipeline uploaded (multiple versions).
2. Create a recurring run, explicitly selecting a specific pipeline version in the UI (not "use latest").
3. Inspect the resulting ScheduledWorkflow CR: kubectl get scheduledworkflow <name> -o yaml.
4. Observe that spec.workflow.spec contains a fully compiled Argo WorkflowSpec, including specific launcher/driver image references, rather than just a pipelineId/pipelineVersionId reference.
5. Upgrade the KFP deployment (new launcher/driver images). Trigger the recurring run again and observe it still uses the old, pre-upgrade images.

Note: this reproduces even with BLOCK_V1_PIPELINES=true, since that flag only blocks V1-templated workflows from executing (shouldEnforceV1Block inspects the compiled workflow for V2 markers) — it does not change which jobs get their manifest embedded vs. referenced.

Proposed fix

Extend the existing reference-based path to cover pinned versions, not just "latest":

1. In the recurring run creation path (resource_manager.go), only resolve-and-embed the manifest when a raw manifest is actually supplied (true legacy/freeform submission). When the request references a stored pipeline version by ID (pinned or not), skip resolution and persist PipelineId/PipelineVersionId on the Job only.
2. No ScheduledWorkflow CRD schema change is required — spec.pipelineId/spec.pipelineVersionId already exist and are already plumbed through to CreateRun's PipelineVersionReference in the controller.
3. ReconcileSwfCrs's existing skip-if-no-manifest check requires no change; it will now also correctly skip pinned-but-referenced jobs.
4. No controller change needed — submitNewWorkflowIfNotAlreadySubmitted already forwards a non-empty PipelineVersionId correctly; it just currently never receives one for pinned jobs because the manifest already exists earlier in the flow.

Net effect: "pin to a specific pipeline version" becomes "always run version X, compiled fresh from the stored version at trigger time" rather than "snapshot version X's compiled output — including images — forever." This removes the CRD/pipeline-version duplication and the image lock-in problem, while preserving the reproducibility guarantee (same pipeline logic) that pinning is meant to provide.

Open questions / things to validate

• Per-trigger template compilation cost for high-frequency pinned schedules (the "latest" path already does this today, so likely acceptable, but worth confirming there's no caching assumption this would violate).
• Backward-compatibility/migration story for existing ScheduledWorkflow CRs that already have a manifest embedded.
• Whether this should be opt-in behind a flag initially, given it's a behavior change for a very common recurring-run pattern.

Environment

• KFP backend: master branch (also reproduces on 2.x releases with BLOCK_V1_PIPELINES=true)
• Confirmed via direct testing: pinning a version through the recurring-run UI bakes the compiled workflow spec into the ScheduledWorkflow CR.