feat(backend): Allow recurring runs to always use the latest pipeline version (kubeflow#11560)

* Allow recurring runs to always use the latest pipeline version

This makes the pipeline_version_id optional and makes the
Scheduled Workflow controller levarege the REST API to launch the run
rather than rely on compiled Argo Workflows stored in the
ScheduledWorkflow object.

The previous behavior is preserved if the user is using the v1 API or
specifies a pipeline version ID or pipeline spec manifest.

Resolves:
kubeflow#11542

Signed-off-by: mprahl <[email protected]>

* Add instructions to debug the SWF controller

Signed-off-by: mprahl <[email protected]>

---------

Signed-off-by: mprahl <[email protected]>

Author: mprahl

.gitignore

@@ -86,5 +86,8 @@ __pycache__
 # kfp local execution default directory
 local_outputs/
 
+# Ignore the Kind cluster kubeconfig
+kubeconfig_dev-pipelines-api
+
 # Ignore debug Driver Dockerfile produced from `make -C backend image_driver_debug`
 backend/Dockerfile.driver-debug

backend/README.md

@@ -159,6 +159,39 @@ You can also directly connect to the MariaDB database server with:
 mysql -h 127.0.0.1 -u root
 ```
 
+### Scheduled Workflow Development
+
+If you also want to run the Scheduled Workflow controller locally, stop the controller on the cluster with:
+
+```bash
+kubectl -n kubeflow scale deployment ml-pipeline-scheduledworkflow --replicas=0
+```
+
+Then you may leverage the following sample `.vscode/launch.json` file to run the Scheduled Workflow controller locally:
+
+```json
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Launch Scheduled Workflow controller (Kind)",
+            "type": "go",
+            "request": "launch",
+            "mode": "debug",
+            "program": "${workspaceFolder}/backend/src/crd/controller/scheduledworkflow",
+            "env": {
+                "CRON_SCHEDULE_TIMEZONE": "UTC"
+            },
+            "args": [
+                "-namespace=kubeflow",
+                "-kubeconfig=${workspaceFolder}/kubeconfig_dev-pipelines-api",
+                "-mlPipelineAPIServerName=localhost"
+            ]
+        }
+    ]
+}
+```
+
 ### Remote Debug the Driver
 
 These instructions assume you are leveraging the Kind cluster in the

backend/api/Makefile

@@ -17,6 +17,8 @@
 IMAGE_TAG=kfp-api-generator
 # Contact chensun or zijianjoy if this remote image needs an update.
 REMOTE_IMAGE=ghcr.io/kubeflow/kfp-api-generator
+# Assume the latest API version by default.
+API_VERSION ?= v2beta1
 
 # Keep in sync with the version used in test/release/Dockerfile.release
 PREBUILT_REMOTE_IMAGE=ghcr.io/kubeflow/kfp-api-generator:1.0

backend/api/README.md

@@ -12,7 +12,7 @@ Tools needed:
 Set the environment variable `API_VERSION` to the version that you want to generate. We use `v1beta1` as example here.
 
 ```bash
-export API_VERSION="v1beta1"
+export API_VERSION="v2beta1"
 ```
 
 ## Compiling `.proto` files to Go client and swagger definitions

backend/api/v1beta1/job.proto

@@ -210,9 +210,9 @@ message Job {
   // Optional input field. Describing the purpose of the job
   string description = 3;
 
-  // Required input field.
+  // Optional input field.
   // Describing what the pipeline manifest and parameters to use
-  // for the scheduled job.
+  // for the scheduled job. If unset, fetch the pipline_spec at runtime.
   PipelineSpec pipeline_spec = 4;
 
   // Optional input field. Specify which resource this job belongs to.