Skip to content

Commit e915bb3

Browse files
author
arpechenin
committed
- fix steps description formatting
- replace links to github code - resolve comments Signed-off-by: arpechenin <[email protected]>
1 parent 1759b86 commit e915bb3

File tree

1 file changed

+35
-28
lines changed
  • proposals/separate-standalone-driver

1 file changed

+35
-28
lines changed

proposals/separate-standalone-driver/README.md

Lines changed: 35 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -25,29 +25,35 @@ Resource availability problems: There's no guarantee the Kubernetes cluster has
2525

2626
Let's take a look at the copy of [hello_world.yaml](hello_world.yaml) generated by the argo compiler tests.
2727

28-
Execution Order:
29-
1. entrypoint
30-
Type: DAG
31-
The root DAG represents the entire pipeline run.
32-
2. root-driver (template: system-dag-driver)
33-
Type: Container
34-
Purpose: Initializes the root DAG. Creates an MLMD execution for the DAG.
35-
Outputs:
36-
execution-id – ID of the DAG execution (created during root-driver execution)
37-
38-
Tasks inside:
39-
- hello-world-driver (template: system-container-driver)
40-
Purpose: Check for the existence of an execution in the cache, if no prepare the MLMD execution of the hello-world container task, and generate the appropriate pod-spec-patch.
41-
Outputs:
42-
pod-spec-patch – patch for the system-container-executor pod; inserts the correct image and command for the main container.
43-
cached-decision – if true, the next step will be skipped
44-
- hello-world (template: system-container-executor)
45-
Depends on: hello-world-driver.Succeeded
46-
Purpose: Executes the hello-world component
47-
Inputs:
48-
pod-spec-patch — patch for the pod generated in the previous step
49-
cached-decision — used as a skip condition
50-
The system-container-executor template defines the main container that runs the user-defined code.
28+
**Execution Order:**
29+
30+
1. **entrypoint**
31+
*Type:* DAG
32+
The root DAG represents the entire pipeline run.
33+
34+
2. **root-driver** *(template: system-dag-driver)*
35+
*Type:* Container
36+
*Purpose:* Initializes the root DAG. Creates an MLMD execution for the DAG.
37+
**Outputs:**
38+
- `execution-id` – ID of the DAG execution (created during root-driver execution)
39+
40+
**Tasks inside:**
41+
42+
- **hello-world-driver** *(template: system-container-driver)*
43+
*Purpose:* Check for the existence of an execution in the cache. If it does not exist, prepare the MLMD execution of the hello-world container task, and generate the appropriate pod-spec-patch.
44+
**Outputs:**
45+
- `pod-spec-patch` – patch for the system-container-executor pod; inserts the correct image and command for the main container
46+
- `cached-decision` – if true, the next step will be skipped
47+
48+
- **hello-world** *(template: system-container-executor)*
49+
*Depends on:* `hello-world-driver.Succeeded`
50+
*Purpose:* Executes the hello-world component
51+
**Inputs:**
52+
- `pod-spec-patch` — patch for the pod generated in the previous step
53+
- `cached-decision` — used as a skip condition
54+
55+
The `system-container-executor` template defines the main container that runs the user-defined code.
56+
5157

5258
Overview of the Argo workflow node structure for the container-driver
5359
```yaml
@@ -70,10 +76,10 @@ Overview of the Argo workflow node structure for the container-driver
7076
```
7177
It creates a pod that launches the driver container using the kfp-driver image.
7278
73-
## Alternative
79+
## Proposal
7480
7581
Instead of launching a new driver's pod using a container template, configure the system to send requests to an already running server.
76-
Something like this (showing both types of containers):
82+
Something like this (showing both types of drivers):
7783
```yaml
7884
templates:
7985
- name: system-container-driver
@@ -145,9 +151,9 @@ Below is a scheme where, instead of creating a pod for the driver's task, we reu
145151
146152
147153
To move from the container template to the Executor Plugin template:
148-
- patch the [Argo compiler](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/compiler/argocompiler) to generate a plugin template instead of a container template. Sample: hello-world [adapted](kfp-plugin-flow.png) (see name: system-container-driver)
149-
- Namely, replace the templates used in the [container-driver](https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/compiler/argocompiler/container.go#L148) and [dag-driver](https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/compiler/argocompiler/dag.go#L156) section of the compiler
150-
- Extract the [driver](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/driver) component into a standalone server.
154+
- patch the [Argo compiler](https://github.com/kubeflow/pipelines/tree/a870b1a325dae0c82c8b6f57941468ee1aea960b/backend/src/v2/compiler/argocompiler) to generate a plugin template instead of a container template. Sample: hello-world [adapted](kfp-plugin-flow.png) (see name: system-container-driver)
155+
- Namely, replace the templates used in the [container-driver](https://github.com/kubeflow/pipelines/blob/a870b1a325dae0c82c8b6f57941468ee1aea960b/backend/src/v2/compiler/argocompiler/container.go#L148) and [dag-driver](https://github.com/kubeflow/pipelines/blob/a870b1a325dae0c82c8b6f57941468ee1aea960b/backend/src/v2/compiler/argocompiler/dag.go#L156) section of the compiler
156+
- Extract the [driver](https://github.com/kubeflow/pipelines/tree/a870b1a325dae0c82c8b6f57941468ee1aea960b/backend/src/v2/driver) component into a standalone server.
151157
- Implement the [plugin](plugin.md)
152158
153159
The sample of the Argo Workflow system-container-driver template based on plugin.
@@ -164,6 +170,7 @@ The sample of the Argo Workflow system-container-driver template based on plugin
164170
- default: ""
165171
name: kubernetes-config
166172
metadata: {}
173+
# this is the key change that specifies use of the executor plugin
167174
plugin:
168175
driver-plugin:
169176
args:

0 commit comments

Comments
 (0)