Add Pipeline design #21
Conversation
Yancey0623
force-pushed
the
pipeline_design
branch
from
b6daece to
e52bd0a
Compare
|
cc @typhoonzero |
|
|
||
| ```python | ||
| skaffold_git = fluid.Git( | ||
| skaffold_git = fluid.git_resource( |
There was a problem hiding this comment.
Please also update below descriptions: Please be aware that the call to fluid.Git doesn't include the name
There was a problem hiding this comment.
Why not use fluid.git for simple?
Just tracing the current implementation. I think we can update the API and design in another PR if need.
There was a problem hiding this comment.
I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.
doc/design.md
Outdated
|
|
||
| ``` python | ||
| @fluid.pipeline | ||
| def tutorial(source_repo:"resource,git", web_image="resource,image"): |
There was a problem hiding this comment.
I think it's okay, this is the parameter annotation instead of the default value, ref: https://www.python.org/dev/peps/pep-3107/#id29
| build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image) | ||
| deploy_web = deploy_using_kubectl(source_repo, web_image) | ||
| deploy_web.web_image.from(build_skaffold_web) |
There was a problem hiding this comment.
How to define dependency by not using input/output?
There was a problem hiding this comment.
Can use the runAfter keyword, and I added a section PiepeLine with DAG to introduce how to construct the DAG.
|
|
||
| ``` yaml | ||
| apiVersion: tekton.dev/v1beta1 | ||
| kind: Pipeline |
There was a problem hiding this comment.
What is the lifecycle of a pipeline object?
There was a problem hiding this comment.
Pipeline object is like function definition in Python which includes Pipeline Tasks
A PipelineRun object would invoke the Pipeline Tasks as the dependency, can find some information from Task Status
|
|
||
| This will result in the following execution graph: | ||
|
|
||
| ``` text |
There was a problem hiding this comment.
No whitespace between ``` and text
|
|
||
| ```python | ||
| skaffold_git = fluid.Git( | ||
| skaffold_git = fluid.git_resource( |
There was a problem hiding this comment.
I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.
|
|
||
| ```python | ||
| skaffold_image_leeroy_web = fluid.Image( | ||
| skaffold_image_leeroy_web = fluid.image_resource( |
There was a problem hiding this comment.
Similarly, "image" is one of a few pre-defined resource types. How about we keep the name Image.
|
|
||
| ```yaml | ||
| goapiVersion: tekton.dev/v1alpha1 | ||
| apiVersion: tekton.dev/v1alpha1 |
There was a problem hiding this comment.
Thanks for pointing this out!
|
|
||
| ### Pipeline | ||
|
|
||
| A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md). |
There was a problem hiding this comment.
In the above text, we stated that a Task is like a function. Here we state the same with Pipeline. What is the difference between these two types of "functions"?
| value: "spec.template.spec.containers[0].image" | ||
| ``` | ||
|
|
||
| The above `Pipeline` is referencing a `Task` called `deploy-using-kubectl` defined as follows: |
There was a problem hiding this comment.
is referencing => refers to
| build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image) | ||
| deploy_web = deploy_using_kubectl(source_repo, web_image) | ||
| deploy_web.web_image.from(build_skaffold_web) |
There was a problem hiding this comment.
This deploy_web.web_image.from syntax looks confusing. Python programmer do not do this with function parameters.
I am afraid that this weird design might come from the fact that a Pipeline is NOT similar to a function definition.
There was a problem hiding this comment.
After reading more about Pipeline, I see it describes a DAG of tasks, where edges are data dependencies between tasks.
Thinking about the following example from the Tekton tutorial https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#from:
- name: build-app
taskRef:
name: build-push
resources:
outputs:
- name: image
resource: my-image
- name: deploy-app
taskRef:
name: deploy-kubectl
resources:
inputs:
- name: image
resource: my-image
from:
- build-appUsing programming language idiom, it is simply function calls
deploy_kubectl(image=build_push(my_image))It seems that what we expect users to write is
@fluid.pipeline
def build_and_deploy(image):
deploy_kubectl(image=build_push(my_image))where @fluid.pipeline should dry-run the function build_and_deploy to analysis the function dependencies, which is deploy_kubectl.image <- build_push, and generate the YAML definition of the Pipeline object.
I am not sure if the above suggestion is correct, or how reasonable it is. It has been a while I haven't use Tekton.
| As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials: | ||
|
|
||
| ``` yaml | ||
| - name: lint-repo |
There was a problem hiding this comment.
This doesn't look like a complete Kubernetes YAML. What is its kind?
| - `from`: clauses on the PipelineResources needed by a Task. | ||
| - `runAfter`: clauses on the Pipeline Tasks. | ||
|
|
||
| As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials: |
There was a problem hiding this comment.
Which tutorial? We need a URL here.
4 participants
Add API design for Tekton Pipeline.