Skip to content

Commit 560864c

Browse files
pritidesaipritidesai
committed
TEP-0147 Dynamic Matrix
Proposing a new section for matrix which allows to specify the explicit list of combinations dyanmically Signed-off-by: Priti Desai <[email protected]>
1 parent 0475e40 commit 560864c

File tree

2 files changed

+379
-0
lines changed

2 files changed

+379
-0
lines changed

‎teps/0147-dynamic-matrix.md‎

Lines changed: 378 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,378 @@
1+
---
2+
status: implementable
3+
title: Dynamic Matrix
4+
creation-date: '2023-10-05'
5+
last-updated: '2023-10-18'
6+
authors: [
7+
'@pritidesai'
8+
]
9+
see-also:
10+
- TEP-0118
11+
- TEP-0090
12+
---
13+
14+
# TEP-0147: Dynamic Matrix
15+
16+
<!-- toc -->
17+
- [Summary](#summary)
18+
- [Motivation](#motivation)
19+
- [Goals](#goals)
20+
- [Non-Goals](#non-goals)
21+
- [Use Cases](#use-cases)
22+
- [Proposal](#proposal)
23+
- [`matrix.strategy` initialized by the `pipelineRun`](#matrixstrategy-initialized-by-the-pipelinerun)
24+
- [`matrix.strategy` specified by a task result](#matrixstrategy-specified-by-a-task-result)
25+
- [GitHub Actions](#github-actions)
26+
- [Future Work](#future-work)
27+
- [References](#references)
28+
<!-- /toc -->
29+
30+
## Summary
31+
32+
Matrix feature of Tekton Pipelines allows pipeline authors to specify multiple concurrent execution of the same task.
33+
The same task can be executed in parallel based on the number of input parameters or a combination of parameters.
34+
35+
Matrix supports specifying [implicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#generating-combinations)
36+
and [explicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#explicit-combinations) combinations of
37+
parameters.
38+
39+
- With implicit combinations, a `pipelineTask` is defined with a list of parameters and one or more values for each
40+
parameter. Tekton controller generates an exhaustive list of combinations based on the specified parameters. Based on
41+
the generated list of combinations, a `pipelineTask` is fanned out and executed with each combination.
42+
43+
```yaml
44+
matrix:
45+
params:
46+
- name: platform
47+
value: $(params.platforms[*])
48+
- name: browser
49+
value: $(params.browsers[*])
50+
...
51+
```
52+
53+
- With explicit combinations, a `pipelineTask` is defined with a list of combinations and Tekton controller fans out the
54+
task based on the number of combinations. Each running instance receives unique combination of input parameters.
55+
56+
```yaml
57+
matrix:
58+
include:
59+
- params:
60+
- name: platform
61+
value: "linux"
62+
- name: browser
63+
value: "chrome"
64+
- params:
65+
- name: platform
66+
value: "mac"
67+
- name: browser
68+
value: "safari"
69+
...
70+
```
71+
72+
Both implicit and explicit combinations fit well with many use cases. At the same time, creating a sharable pipeline with
73+
existing `matrix` syntax is not possible when a list of combinations are specified at the runtime through `pipelineRun`.
74+
75+
```yaml
76+
apiVersion: tekton.dev/v1
77+
kind: Pipeline
78+
metadata:
79+
name: cd-pipeline
80+
spec:
81+
workspaces:
82+
- shared-workspace
83+
params:
84+
- name: image-1
85+
- name: image-2
86+
- name: image-3
87+
- name: dockerfile-1
88+
- name: dockerfile-2
89+
- name: dockerfile-3
90+
tasks:
91+
- name: build
92+
taskRef:
93+
name: kaniko
94+
workspaces:
95+
- name: source
96+
workspace: shared-workspace
97+
matrix:
98+
include:
99+
- name: build-1
100+
params:
101+
- name: IMAGE
102+
value: $(params.image-1)
103+
- name: DOCKERFILE
104+
value: $(params.dockerfile-1)
105+
- name: build-2
106+
params:
107+
- name: IMAGE
108+
value: $(params.image-2)
109+
- name: DOCKERFILE
110+
value: $(params.dockerfile-2)
111+
- name: build-3
112+
params:
113+
- name: IMAGE
114+
value: $(params.image-3)
115+
- name: DOCKERFILE
116+
value: $(params.dockerfile-3)
117+
---
118+
119+
apiVersion: tekton.dev/v1beta1
120+
kind: PipelineRun
121+
metadata:
122+
generateName: cd-pipelinerun-
123+
spec:
124+
pipelineRef:
125+
name: cd-pipeline
126+
params:
127+
- name: image-1
128+
value: "image-1"
129+
- name: dockerfile-1
130+
value: "path/to/1/Dockerfile"
131+
- name: image-2
132+
value: "image-2"
133+
- name: dockerfile-2
134+
value: "path/to/2/Dockerfile"
135+
- name: image-3
136+
value: "image-3"
137+
- name: dockerfile-3
138+
value: "path/to/3/Dockerfile"
139+
```
140+
141+
## Motivation
142+
143+
It is a common practice to execute a `pipeline` either by creating a `PipelineRun` object or through `TriggerTemplate`.
144+
Generally, a list of `params` are specified as part of the `PipelineRun` which might be initialized statically or
145+
through a [trigger template params](https://github.com/tektoncd/triggers/blob/main/docs/triggertemplates.md#structure-of-a-triggertemplate).
146+
Now, running a shared `pipeline` with `matrix` where the values for each instance of fanned out task is dynamically specified
147+
at the runtime is not possible with the existing `matrix` syntax. In this proposal, we would like to extend `matrix`
148+
syntax to provide an option to dynamically specify a list of explicit combinations
149+
150+
### Goals
151+
152+
- Be able to author a sharable pipeline with `matrix` such that a task can fan out based on the explicit combinations
153+
which are specified dynamically.
154+
155+
### Non-Goals
156+
157+
- Matrix explicit combination mechanism only supports a `param` of type `string` and not adding support for any
158+
other type.
159+
160+
### Use Cases
161+
162+
Let's revisit the example of build pipeline from [TEP-0118](0118-matrix-with-explicit-combinations-of-parameters.md#define-explicit-combinations-in-the-matrix).
163+
A pipeline `pipeline-to-build-images-from-a-single-repo` has three explicit combinations defined in the pipeline such
164+
that `kaniko` can fan out and create three separate `taskRun`s. The values for each combination can be furthered
165+
dynamically specified using the `params` as seen in the example in the [summary](#summary).
166+
167+
```yaml
168+
apiVersion: tekton.dev/v1beta1
169+
kind: Pipeline
170+
metadata:
171+
name: pipeline-to-build-images-from-a-single-repo
172+
spec:
173+
workspaces:
174+
- name: shared-workspace
175+
tasks:
176+
- ...
177+
- name: kaniko-build
178+
taskRef:
179+
name: kaniko
180+
workspaces:
181+
- name: source
182+
workspace: shared-workspace
183+
matrix:
184+
include:
185+
- name: build-1
186+
params:
187+
- name: IMAGE
188+
value: "image-1"
189+
- name: DOCKERFILE
190+
value: "path/to/Dockerfile1"
191+
- name: build-2
192+
params:
193+
- name: IMAGE
194+
value: "image-2"
195+
- name: DOCKERFILE
196+
value: "path/to/Dockerfile2"
197+
- name: build-3
198+
params:
199+
- name: IMAGE
200+
value: "image-3"
201+
- name: DOCKERFILE
202+
value: "path/to/Dockerfile3"
203+
- ...
204+
```
205+
206+
This `pipeline` is part of a catalog and shared across multiple teams. Now, the teams are building their own applications
207+
using this `pipeline` and different teams have different number of images to built as part of their application. This way
208+
of specifying `matrix` combinations in a `pipeline` is constant and can not be utilized for an application with
209+
different number of images.
210+
211+
## Proposal
212+
213+
We propose adding a field - `strategy` of type `string` - within the `matrix` section. This allows pipeline authors to
214+
maintain a single pipeline in a catalog and allow the users to specify the list of explicit combinations for each
215+
instance dynamically through `pipelineRun` or a trigger template or a task result.
216+
217+
The `matrix.strategy` is of type `string` where its value is a list of objects in key:value pairs. The controller reads
218+
a stringified JSON payload in a form of `{"include": a list of combinations}` and creates an equivalent number of
219+
instances based on the length of the specified list.
220+
221+
When `matrix.strategy` is specified, no other `matrix` fields will be allowed i.e. `matrix.params` and `matrix.include`
222+
are not permitted with `matrix.strategy`. This restriction will help avoid any undesired conflicts after the `matrix`
223+
is resolved.
224+
225+
**NOTE:** `matrix.include` support an optional field `name`. `name` is for information purpose only and has no real
226+
significance in how combinations are being generated. `name` is useful from the readability perspective with
227+
the explicit combinations when specified in the pipeline definition itself. With this proposal, `name` is not included.
228+
The reason behind not includeing `name` is, JSON specifications inherently does not support an optional fields.
229+
The users have to specify empty string at the minimum.
230+
231+
### `matrix.strategy` initialized by the `pipelineRun`
232+
233+
The `Pipeline` addressing this use case will be defined as shown below:
234+
235+
```yaml
236+
apiVersion: tekton.dev/v1beta1
237+
kind: Pipeline
238+
metadata:
239+
name: pipeline-to-build-images-from-a-single-repo
240+
spec:
241+
params:
242+
- name: matrix-strategy
243+
workspaces:
244+
- name: shared-workspace
245+
tasks:
246+
- ...
247+
- name: kaniko-build
248+
taskRef:
249+
name: kaniko
250+
workspaces:
251+
- name: source
252+
workspace: shared-workspace
253+
matrix:
254+
strategy: $(params.matrix-strategy)
255+
```
256+
257+
A `PipelineRun` can be created to execute the above `pipeline` to build 1 image:
258+
259+
```yaml
260+
apiVersion: tekton.dev/v1beta1
261+
kind: PipelineRun
262+
metadata:
263+
generateName: pipelineRun-
264+
spec:
265+
pipelineRef:
266+
name: pipeline-to-build-images-from-a-single-repo
267+
params:
268+
- name: matrix-strategy
269+
value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}"
270+
```
271+
272+
`PipelineRun` can be created to execute the above `pipeline` to build 3 images:
273+
274+
```yaml
275+
apiVersion: tekton.dev/v1beta1
276+
kind: PipelineRun
277+
metadata:
278+
generateName: pipelineRun-
279+
spec:
280+
pipelineRef:
281+
name: pipeline-to-build-images-from-a-single-repo
282+
params:
283+
- name: matrix-strategy
284+
value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"},{\"IMAGE\":\"image-2\",\"DOCKERFILE\":\"path/to/Dockerfile2\"},{\"IMAGE\":\"image-3\",\"DOCKERFILE\":\"path/to/Dockerfile3\"}]}"
285+
```
286+
287+
### `matrix.strategy` specified by a task result
288+
289+
```yaml
290+
apiVersion: tekton.dev/v1beta1
291+
kind: Pipeline
292+
metadata:
293+
name: pipeline-to-build-images-from-a-single-repo
294+
spec:
295+
workspaces:
296+
- name: shared-workspace
297+
tasks:
298+
- ...
299+
- name: pre-requisite
300+
taskSpec:
301+
results:
302+
- name: matrix-strategy
303+
steps:
304+
- name: read-strategy
305+
image: alpine
306+
script: |
307+
# this strategy can be read from a JSON file in the repo
308+
echo -n "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}" | tee $(results.matrix-strategy.path)
309+
- name: kaniko-build
310+
params:
311+
- name: matrix-strategy
312+
value: $(tasks.read-matrix-strategy.results.matrix-strategy)
313+
taskRef:
314+
name: kaniko
315+
workspaces:
316+
- name: source
317+
workspace: shared-workspace
318+
matrix:
319+
strategy: $(params.matrix-strategy)
320+
---
321+
322+
apiVersion: tekton.dev/v1beta1
323+
kind: PipelineRun
324+
metadata:
325+
generateName: pipelineRun-
326+
spec:
327+
pipelineRef:
328+
name: pipeline-to-build-images-from-a-single-repo
329+
```
330+
331+
### GitHub Actions
332+
333+
This syntax is inspired by the GitHub actions syntax similar to the rest of the sections of `matrix`.
334+
335+
[GitHub actions](https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object)
336+
does support specifying a list of jobs in which a JSON payload is set in one job and passed to the next job using
337+
`output` and `fromJSON`.
338+
339+
```yaml
340+
name: build
341+
on: push
342+
jobs:
343+
job1:
344+
runs-on: ubuntu-latest
345+
outputs:
346+
matrix: ${{ steps.set-matrix.outputs.matrix }}
347+
steps:
348+
- id: set-matrix
349+
run: echo "matrix={\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" >> $GITHUB_OUTPUT
350+
job2:
351+
needs: job1
352+
runs-on: ubuntu-latest
353+
strategy:
354+
matrix: ${{ fromJSON(needs.job1.outputs.matrix) }}
355+
steps:
356+
- run: build
357+
```
358+
359+
## Alternatives
360+
361+
- A feature is requested in general to support [nested parameters](https://github.com/tektoncd/pipeline/issues/7069).
362+
The idea here is to be able to specify array and object params within array and object params. This
363+
feature can help support [dynamic specifications](https://github.com/tektoncd/pipeline/issues/7069#issuecomment-1721670122)
364+
by using an array of object i.e. a list of key value pairs.
365+
366+
## Future Work
367+
368+
- The proposal here is to support a string version of JSON object right now but this can be extended in future to support
369+
an expression language such as CEL. The current proposal does not support an expression language but provides an
370+
opportunity to extend the existing proposal.
371+
372+
## References
373+
374+
- [#7170]: https://github.com/tektoncd/pipeline/issues/7170
375+
- [Tekton Pipelines PoC]: https://github.com/tektoncd/pipeline/pull/7180
376+
- [Dynamic Matrix in GitHub Actions]: https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions
377+
- [GitHub Actions Matrix]: https://adamtheautomator.com/github-actions-matrix/
378+
- [Nested Params]: https://github.com/tektoncd/pipeline/issues/7069

‎teps/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,3 +131,4 @@ This is the complete list of Tekton TEPs:
131131
|[TEP-0140](0140-producing-results-in-matrix.md) | Producing Results in Matrix | implementable | 2023-08-21 |
132132
|[TEP-0141](0141-platform-context-variables.md) | Platform Context Variables | proposed | 2023-08-21 |
133133
|[TEP-0145](0145-cel-in-whenexpression.md) | CEL in WhenExpression | implementable | 2023-10-01 |
134+
|[TEP-0147](0147-dynamic-matrix.md) | Dynamic Matrix | implementable | 2023-10-18 |

0 commit comments

Comments
 (0)