Add details

Signed-off-by: arpechenin <[email protected]>

Author: arpechenin

proposals/separate-standalone-driver/README.md

@@ -125,10 +125,9 @@ Below is a scheme where, instead of creating a pod for the driver's task, we reu
 
 To move from the container template to the Executor Plugin template:
 - patch the [Argo compiler](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/compiler/argocompiler) to generate a plugin template instead of a container template. Sample: hello-world [adapted](kfp-plugin-flow.png) (see name: system-container-driver)
-- Implement the [driver](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/driver) as a plugin - that is, as a  HTTP Server which implements the expected [contract](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#example-a-simple-python-plugin).
-  This server will run as a sidecar container alongside the agent pod.
-- Enable plugin extensions in the workflow-controller. See the [Configuration](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#configuration) guide.
-  Then, [install]([Configuration](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#configuration)) the plugin built in the previous step into the workflow-controller.
+- Namely, replace the templates used in the [container-driver](https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/compiler/argocompiler/container.go#L148) and [dag-driver](https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/compiler/argocompiler/dag.go#L156) section of the compiler
+- Extract the [driver](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/driver) component into a standalone server.
+- Implement the [plugin](plugin.md)
 
 The sample of the Argo Workflow system-container-driver template based on plugin.
 ```yaml
@@ -175,11 +174,6 @@ The sample of the Argo Workflow system-container-driver template based on plugin
               default: "true"
               jsonPath: $.condition
 ```
-The `driver-plugin` section refers to the name under which the standalone driver was installed, as described in the  [plugin](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#example-a-simple-python-plugin)
-```yaml
-plugin:
-    driver-plugin:
-```
 
 ## Conclusion
 This proposal introduces an optimization for Kubeflow Pipelines (KFP) that replaces per-task driver pods with a lightweight standalone service based on Argo Workflows’ Executor Plugin mechanism. It significantly reduces pipeline task startup time by eliminating the overhead of scheduling a separate driver pod for each task — particularly beneficial for large pipelines with multiple steps and caching enabled.

proposals/separate-standalone-driver/plugin.md

@@ -0,0 +1,27 @@
+## Mock Implementation of Argo Workflow Executor Plugin
+
+According to the [documentation](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/), an Argo Workflow plugin is a sidecar container that runs in the agent pod. The agent pod is created once for each workflow.
+
+### terminology
+- kfp-driver-server — the KFP [driver](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/driver) component extracted from the workflow pod and deployed as a global HTTP/gRPC server.
+- driver-plugin - our implementation of the Executor plugin
+
+*One limitation is that the service account token is not mounted into the sidecar container, which means it cannot interact with the Kubernetes API. This is required for drivers.*
+As a result, driver-plugin implementations should merely act as a proxy between the global kfp-driver-server and the Argo Workflow controller.
+
+### Prerequisites
+- According to the [configuration](https://argo-workflows.readthedocs.io/en/latest/executor_plugins/#configuration) ARGO_EXECUTOR_PLUGINS should be set to true 
+- Add additional [workflow RBAC](https://argo-workflows.readthedocs.io/en/latest/http-template/#argo-agent-rbac) for the agent
+
+1. Implement the driver plugin that simply proxies requests from the workflow controller to the kfp-driver-server and back. Check the mock [implementation](src/driver-plugin)
+2. Build the image for the driver plugin.
+3. Create the [yaml description](src/driver-plugin/plugin.yaml) of the plugin 
+4. [Create](https://argo-workflows.readthedocs.io/en/latest/cli/argo_executor-plugin_build/) the configmap by executing  ```argo executor-plugin build .``` in the yaml description folder from the step 3 
+5. Apply the created ConfigMap to the workflow-controller Kubernetes namespace.
+
+After that, you will be able to reference the corresponding driver plugin in your Argo Workflow using:
+```yaml
+plugin:
+    driver-plugin:
+    ...
+```

proposals/separate-standalone-driver/src/driver-plugin/Dockerfile

@@ -0,0 +1,5 @@
+FROM python:3.12
+WORKDIR /app
+COPY . .
+RUN pip install -r requirements.txt
+CMD ["python", "main.py"]

proposals/separate-standalone-driver/src/driver-plugin/main.py

@@ -0,0 +1,32 @@
+from fastapi import FastAPI, Request
+import uvicorn
+import aiohttp
+
+app = FastAPI()
+
+
+async def call_driver(url, payload):
+    async with aiohttp.ClientSession() as session:
+        async with session.post(url, json=payload) as response:
+            if response.status != 200:
+                text = await response.text()
+                raise Exception(f"driver call failed with status: {response.status} error: {text}")
+            content_type = response.headers.get("Content-Type", "")
+            if "application/json" not in content_type:
+                text = await response.text()
+                raise Exception(f"driver returns unexpected Content-Type: {content_type}, response: {text}")
+            return await response.json()
+
+
+@app.post("/api/v1/template.execute")
+async def execute_plugin(request: Request):
+    body = await request.json()
+    payload = body.get("template", {}).get("plugin", {}).get("driver-plugin", {}).get("args", {})
+    print("request payload:" + str(payload))
+    response = await call_driver("http://ml-pipeline-kfp-driver.kubeflow.svc.cluster.local:2948/api/v1/execute", payload)
+    print("response:", response)
+    return response
+
+
+if __name__ == "__main__":
+    uvicorn.run("main:app", host="0.0.0.0", port=2948, reload=True)

proposals/separate-standalone-driver/src/driver-plugin/plugin.yaml

@@ -0,0 +1,21 @@
+apiVersion: argoproj.io/v1alpha1
+kind: ExecutorPlugin
+metadata:
+  name: driver-plugin
+spec:
+  sidecar:
+    container:
+      image: <YOUR_IMAGE>
+      name: driver-plugin
+      ports:
+        - containerPort: 2948
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+      resources:
+        requests:
+          memory: "512Mi"
+          cpu: "250m"
+        limits:
+          memory: "1Gi"
+          cpu: "1"

proposals/separate-standalone-driver/src/driver-plugin/requirements.txt

@@ -0,0 +1,3 @@
+fastapi
+aiohttp
+uvicorn[standard]