Skip to content

Commit 351b457

Browse files
pritidesaipritidesai
committed
TEP-0147 Dynamic Matrix
Proposing a new section for matrix which allows to specify the explicit list of combinations dyanmically Signed-off-by: Priti Desai <pdesai@us.ibm.com>
1 parent f069a45 commit 351b457

File tree

2 files changed

+362
-0
lines changed

2 files changed

+362
-0
lines changed

‎teps/0147-dynamic-matrix.md‎

Lines changed: 361 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,361 @@
1+
---
2+
status: implementable
3+
title: Dynamic Matrix
4+
creation-date: '2023-10-05'
5+
last-updated: '2023-10-12'
6+
authors: [
7+
'@pritidesai'
8+
]
9+
see-also:
10+
- TEP-0118
11+
- TEP-0090
12+
---
13+
14+
# TEP-0147: Dynamic Matrix
15+
16+
<!-- toc -->
17+
- [Summary](#summary)
18+
- [Motivation](#motivation)
19+
- [Goals](#goals)
20+
- [Non-Goals](#non-goals)
21+
- [Use Cases](#use-cases)
22+
- [Proposal](#proposal)
23+
- [`matrix.strategy` initialized by the `pipelineRun`](#matrixstrategy-initialized-by-the-pipelinerun)
24+
- [`matrix.strategy` specified by a task result](#matrixstrategy-specified-by-a-task-result)
25+
- [GitHub Actions](#github-actions)
26+
- [References](#references)
27+
<!-- /toc -->
28+
29+
## Summary
30+
31+
Matrix feature of Tekton Pipelines allows pipeline authors to specify multiple concurrent execution of the same task.
32+
The same task can be executed in parallel based on the number of input parameters or a combination of parameters.
33+
34+
Matrix supports specifying [implicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#generating-combinations)
35+
and [explicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#explicit-combinations) combinations of
36+
parameters.
37+
38+
- With implicit combinations, a `pipelineTask` is defined with a list of parameters and one or more values for each
39+
parameter. Tekton controller generates an exhaustive list of combinations based on the specified parameters. Based on
40+
the generated list of combinations, a `pipelineTask` is fanned out and executed with each combination.
41+
42+
```yaml
43+
matrix:
44+
params:
45+
- name: platform
46+
value: $(params.platforms[*])
47+
- name: browser
48+
value: $(params.browsers[*])
49+
...
50+
```
51+
52+
- With explicit combinations, a `pipelineTask` is defined with a list of combinations and Tekton controller fans out the
53+
task based on the number of combinations. Each running instance receives unique combination of input parameters.
54+
55+
```yaml
56+
matrix:
57+
include:
58+
- params:
59+
- name: platform
60+
value: "linux"
61+
- name: browser
62+
value: "chrome"
63+
- params:
64+
- name: platform
65+
value: "mac"
66+
- name: browser
67+
value: "safari"
68+
...
69+
```
70+
71+
Both implicit and explicit combinations fit well with many use cases. At the same time, creating a sharable pipeline with
72+
existing `matrix` syntax is not possible when a list of combinations are specified at the runtime through `pipelineRun`.
73+
74+
```yaml
75+
apiVersion: tekton.dev/v1
76+
kind: Pipeline
77+
metadata:
78+
name: cd-pipeline
79+
spec:
80+
workspaces:
81+
- shared-workspace
82+
params:
83+
- name: image-1
84+
- name: image-2
85+
- name: image-3
86+
- name: dockerfile-1
87+
- name: dockerfile-2
88+
- name: dockerfile-3
89+
tasks:
90+
- name: build
91+
taskRef:
92+
name: kaniko
93+
workspaces:
94+
- name: source
95+
workspace: shared-workspace
96+
matrix:
97+
include:
98+
- name: build-1
99+
params:
100+
- name: IMAGE
101+
value: $(params.image-1)
102+
- name: DOCKERFILE
103+
value: $(params.dockerfile-1)
104+
- name: build-2
105+
params:
106+
- name: IMAGE
107+
value: $(params.image-2)
108+
- name: DOCKERFILE
109+
value: $(params.dockerfile-2)
110+
- name: build-3
111+
params:
112+
- name: IMAGE
113+
value: $(params.image-3)
114+
- name: DOCKERFILE
115+
value: $(params.dockerfile-3)
116+
---
117+
118+
apiVersion: tekton.dev/v1beta1
119+
kind: PipelineRun
120+
metadata:
121+
generateName: cd-pipelinerun-
122+
spec:
123+
pipelineRef:
124+
name: cd-pipeline
125+
params:
126+
- name: image-1
127+
value: "image-1"
128+
- name: dockerfile-1
129+
value: "path/to/1/Dockerfile"
130+
- name: image-2
131+
value: "image-2"
132+
- name: dockerfile-2
133+
value: "path/to/2/Dockerfile"
134+
- name: image-3
135+
value: "image-3"
136+
- name: dockerfile-3
137+
value: "path/to/3/Dockerfile"
138+
```
139+
140+
## Motivation
141+
142+
It is a common practice to execute a `pipeline` either by creating a `PipelineRun` object or through `TriggerTemplate`.
143+
Generally, a list of `params` are specified as part of the `PipelineRun` which might be initialized statically or
144+
through a [trigger template params](https://github.com/tektoncd/triggers/blob/main/docs/triggertemplates.md#structure-of-a-triggertemplate).
145+
Now, running a shared `pipeline` with `matrix` where the values for each instance of fanned out task is dynamically specified
146+
at the runtime is not possible with the existing `matrix` syntax. In this proposal, we would like to extend `matrix`
147+
syntax to provide an option to dynamically specify a list of explicit combinations
148+
149+
### Goals
150+
151+
- Be able to author a sharable pipeline with `matrix` such that a task can fan out based on the explicit combinations
152+
which are specified dynamically.
153+
154+
### Non-Goals
155+
156+
- Matrix explicit combination mechanism only supports a `param` of type `string` and not adding support for any
157+
other type.
158+
159+
### Use Cases
160+
161+
Let's revisit the example of build pipeline from [TEP-0118](0118-matrix-with-explicit-combinations-of-parameters.md#define-explicit-combinations-in-the-matrix).
162+
A pipeline `pipeline-to-build-images-from-a-single-repo` has three explicit combinations defined in the pipeline such
163+
that `kaniko` can fan out and create three separate `taskRun`s. The values for each combination can be furthered
164+
dynamically specified using the `params` as seen in the example in the [summary](#summary).
165+
166+
```yaml
167+
apiVersion: tekton.dev/v1beta1
168+
kind: Pipeline
169+
metadata:
170+
name: pipeline-to-build-images-from-a-single-repo
171+
spec:
172+
workspaces:
173+
- name: shared-workspace
174+
tasks:
175+
- ...
176+
- name: kaniko-build
177+
taskRef:
178+
name: kaniko
179+
workspaces:
180+
- name: source
181+
workspace: shared-workspace
182+
matrix:
183+
include:
184+
- name: build-1
185+
params:
186+
- name: IMAGE
187+
value: "image-1"
188+
- name: DOCKERFILE
189+
value: "path/to/Dockerfile1"
190+
- name: build-2
191+
params:
192+
- name: IMAGE
193+
value: "image-2"
194+
- name: DOCKERFILE
195+
value: "path/to/Dockerfile2"
196+
- name: build-3
197+
params:
198+
- name: IMAGE
199+
value: "image-3"
200+
- name: DOCKERFILE
201+
value: "path/to/Dockerfile3"
202+
- ...
203+
```
204+
205+
This `pipeline` is part of a catalog and shared across multiple teams. Now, the teams are building their own applications
206+
using this `pipeline` and different teams have different number of images to built as part of their application. This way
207+
of specifying `matrix` combinations in a `pipeline` is constant and can not be utilized for an application with
208+
different number of images.
209+
210+
## Proposal
211+
212+
We propose adding a field - `strategy` of type `string` - within the `matrix` section. This allows pipeline authors to
213+
maintain a single pipeline in a catalog and allow the users to specify the list of explicit combinations for each
214+
instance dynamically through `pipelineRun` or a trigger template or a task result.
215+
216+
The `matrix.strategy` is of type `string` where its value is a list of objects in key:value pairs. The controller reads
217+
a stringified JSON payload in a form of `{"include": a list of combinations}` and creates an equivalent number of
218+
instances based on the length of the specified list.
219+
220+
When `matrix.strategy` is specified, no other `matrix` fields will be allowed i.e. `matrix.params` and `matrix.include`
221+
are not permitted with `matrix.strategy`. This restriction will help avoid any undesired conflicts after the `matrix`
222+
is resolved.
223+
224+
### `matrix.strategy` initialized by the `pipelineRun`
225+
226+
The `Pipeline` addressing this use case will be defined as shown below:
227+
228+
```yaml
229+
apiVersion: tekton.dev/v1beta1
230+
kind: Pipeline
231+
metadata:
232+
name: pipeline-to-build-images-from-a-single-repo
233+
spec:
234+
params:
235+
- name: matrix-strategy
236+
workspaces:
237+
- name: shared-workspace
238+
tasks:
239+
- ...
240+
- name: kaniko-build
241+
taskRef:
242+
name: kaniko
243+
workspaces:
244+
- name: source
245+
workspace: shared-workspace
246+
matrix:
247+
strategy: $(params.matrix-strategy)
248+
```
249+
250+
A `PipelineRun` can be created to execute the above `pipeline` to build 1 image:
251+
252+
```yaml
253+
apiVersion: tekton.dev/v1beta1
254+
kind: PipelineRun
255+
metadata:
256+
generateName: pipelineRun-
257+
spec:
258+
pipelineRef:
259+
name: pipeline-to-build-images-from-a-single-repo
260+
params:
261+
- name: matrix-strategy
262+
value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}"
263+
```
264+
265+
`PipelineRun` can be created to execute the above `pipeline` to build 3 images:
266+
267+
```yaml
268+
apiVersion: tekton.dev/v1beta1
269+
kind: PipelineRun
270+
metadata:
271+
generateName: pipelineRun-
272+
spec:
273+
pipelineRef:
274+
name: pipeline-to-build-images-from-a-single-repo
275+
params:
276+
- name: matrix-strategy
277+
value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"},{\"IMAGE\":\"image-2\",\"DOCKERFILE\":\"path/to/Dockerfile2\"},{\"IMAGE\":\"image-3\",\"DOCKERFILE\":\"path/to/Dockerfile3\"}]}"
278+
```
279+
280+
### `matrix.strategy` specified by a task result
281+
282+
```yaml
283+
apiVersion: tekton.dev/v1beta1
284+
kind: Pipeline
285+
metadata:
286+
name: pipeline-to-build-images-from-a-single-repo
287+
spec:
288+
workspaces:
289+
- name: shared-workspace
290+
tasks:
291+
- ...
292+
- name: pre-requisite
293+
taskSpec:
294+
results:
295+
- name: matrix-strategy
296+
steps:
297+
- name: read-strategy
298+
image: alpine
299+
script: |
300+
# this strategy can be read from a JSON file in the repo
301+
echo -n "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}" | tee $(results.matrix-strategy.path)
302+
- name: kaniko-build
303+
params:
304+
- name: matrix-strategy
305+
value: $(tasks.read-matrix-strategy.results.matrix-strategy)
306+
taskRef:
307+
name: kaniko
308+
workspaces:
309+
- name: source
310+
workspace: shared-workspace
311+
matrix:
312+
strategy: $(params.matrix-strategy)
313+
---
314+
315+
apiVersion: tekton.dev/v1beta1
316+
kind: PipelineRun
317+
metadata:
318+
generateName: pipelineRun-
319+
spec:
320+
pipelineRef:
321+
name: pipeline-to-build-images-from-a-single-repo
322+
```
323+
324+
### GitHub Actions
325+
326+
This syntax is inspired by the GitHub actions syntax similar to the rest of the sections of `matrix`.
327+
328+
[GitHub actions](https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object)
329+
does support specifying a list of jobs in which a JSON payload is set in one job and passed to the next job using
330+
`output` and `fromJSON`.
331+
332+
```yaml
333+
name: build
334+
on: push
335+
jobs:
336+
job1:
337+
runs-on: ubuntu-latest
338+
outputs:
339+
matrix: ${{ steps.set-matrix.outputs.matrix }}
340+
steps:
341+
- id: set-matrix
342+
run: echo "matrix={\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" >> $GITHUB_OUTPUT
343+
job2:
344+
needs: job1
345+
runs-on: ubuntu-latest
346+
strategy:
347+
matrix: ${{ fromJSON(needs.job1.outputs.matrix) }}
348+
steps:
349+
- run: build
350+
```
351+
352+
The proposal here is to support a string version of JSON object right now but this can be extended in future to support
353+
an expression language such as CEL. The current proposal does not bind us to any expression language but provides an
354+
opportunity to support additional expression language along with a JSON payload.
355+
356+
## References
357+
358+
- [#7170]: https://github.com/tektoncd/pipeline/issues/7170
359+
- [Tekton Pipelines PoC]: https://github.com/tektoncd/pipeline/pull/7180
360+
- [Dynamic Matrix in GitHub Actions]: https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions
361+
- [GitHub Actions Matrix]: https://adamtheautomator.com/github-actions-matrix/

‎teps/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,3 +130,4 @@ This is the complete list of Tekton TEPs:
130130
|[TEP-0138](0138-decouple-api-and-feature-versioning.md) | Decouple api and feature versioning | proposed | 2023-07-27 |
131131
|[TEP-0140](0140-producing-results-in-matrix.md) | Producing Results in Matrix | implementable | 2023-08-21 |
132132
|[TEP-0141](0141-platform-context-variables.md) | Platform Context Variables | proposed | 2023-08-21 |
133+
|[TEP-0147](0147-dynamic-matrix.md) | Dynamic Matrix | implementable | 2023-10-12 |

0 commit comments

Comments
 (0)