Skip to content

Commit 4454646

Browse files
crazy-maxcrazy-max
committed
build: reusable workflows to distribute multi-platform build
Signed-off-by: CrazyMax <[email protected]>
1 parent 91676d3 commit 4454646

File tree

1 file changed

+71
-247
lines changed

1 file changed

+71
-247
lines changed

‎content/manuals/build/ci/github-actions/multi-platform.md

Lines changed: 71 additions & 247 deletions
Original file line numberDiff line numberDiff line change
@@ -98,142 +98,63 @@ jobs:
9898
tags: user/app:latest
9999
```
100100

101-
## Distribute build across multiple runners
101+
## Distribute multi-platform build across runners
102102

103103
In the previous example, each platform is built on the same runner which can
104104
take a long time depending on the number of platforms and your Dockerfile.
105105

106-
To solve this issue you can use a matrix strategy to distribute the build for
107-
each platform across multiple runners and create manifest list using the
108-
[`buildx imagetools create` command](/reference/cli/docker/buildx/imagetools/create.md).
106+
To solve this issue, we have created reusable workflows for both
107+
[`docker/build-push-action`](https://github.com/docker/build-push-action/blob/v6/.github/workflows/reusable-distribute-mp.yml)
108+
and [`docker/bake-action`](https://github.com/docker/bake-action/blob/v6/.github/workflows/reusable-distribute-mp.yml)
109+
actions to distribute multi-platform builds across runners efficiently.
109110

110-
The following workflow will build the image for each platform on a dedicated
111-
runner using a matrix strategy and push by digest. Then, the `merge` job will
112-
create manifest lists and push them to two registries:
111+
{{< tabs >}}
112+
{{< tab name="build-push-action" >}}
113113

114-
- Docker Hub: `docker.io/docker-user/my-app`
115-
- GitHub Container Registry: `ghcr.io/gh-user/my-app`
116-
117-
This example also uses the [`metadata` action](https://github.com/docker/metadata-action)
118-
to set tags and labels.
119-
120-
```yaml
114+
```yaml {hl_lines=9}
121115
name: ci
122116
123117
on:
124118
push:
125-
126-
env:
127-
REGISTRY_IMAGE: user/app
119+
pull_request:
128120
129121
jobs:
130122
build:
131-
runs-on: ubuntu-latest
132-
strategy:
133-
fail-fast: false
134-
matrix:
135-
platform:
136-
- linux/amd64
137-
- linux/arm64
138-
steps:
139-
- name: Prepare
140-
run: |
141-
platform=${{ matrix.platform }}
142-
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
143-
144-
- name: Docker meta
145-
id: meta
146-
uses: docker/metadata-action@v5
147-
with:
148-
images: ${{ env.REGISTRY_IMAGE }}
149-
150-
- name: Login to Docker Hub
151-
uses: docker/login-action@v3
152-
with:
153-
username: ${{ vars.DOCKERHUB_USERNAME }}
154-
password: ${{ secrets.DOCKERHUB_TOKEN }}
155-
156-
- name: Set up QEMU
157-
uses: docker/setup-qemu-action@v3
158-
159-
- name: Set up Docker Buildx
160-
uses: docker/setup-buildx-action@v3
161-
162-
- name: Build and push by digest
163-
id: build
164-
uses: docker/build-push-action@v6
165-
with:
166-
platforms: ${{ matrix.platform }}
167-
labels: ${{ steps.meta.outputs.labels }}
168-
tags: ${{ env.REGISTRY_IMAGE }}
169-
outputs: type=image,push-by-digest=true,name-canonical=true,push=true
170-
171-
- name: Export digest
172-
run: |
173-
mkdir -p ${{ runner.temp }}/digests
174-
digest="${{ steps.build.outputs.digest }}"
175-
touch "${{ runner.temp }}/digests/${digest#sha256:}"
176-
177-
- name: Upload digest
178-
uses: actions/upload-artifact@v4
179-
with:
180-
name: digests-${{ env.PLATFORM_PAIR }}
181-
path: ${{ runner.temp }}/digests/*
182-
if-no-files-found: error
183-
retention-days: 1
184-
185-
merge:
186-
runs-on: ubuntu-latest
187-
needs:
188-
- build
189-
steps:
190-
- name: Download digests
191-
uses: actions/download-artifact@v4
192-
with:
193-
path: ${{ runner.temp }}/digests
194-
pattern: digests-*
195-
merge-multiple: true
196-
197-
- name: Login to Docker Hub
198-
uses: docker/login-action@v3
199-
with:
200-
username: ${{ vars.DOCKERHUB_USERNAME }}
201-
password: ${{ secrets.DOCKERHUB_TOKEN }}
202-
203-
- name: Set up Docker Buildx
204-
uses: docker/setup-buildx-action@v3
205-
206-
- name: Docker meta
207-
id: meta
208-
uses: docker/metadata-action@v5
209-
with:
210-
images: ${{ env.REGISTRY_IMAGE }}
211-
tags: |
212-
type=ref,event=branch
213-
type=ref,event=pr
214-
type=semver,pattern={{version}}
215-
type=semver,pattern={{major}}.{{minor}}
216-
217-
- name: Create manifest list and push
218-
working-directory: ${{ runner.temp }}/digests
219-
run: |
220-
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
221-
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
222-
223-
- name: Inspect image
224-
run: |
225-
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}
123+
uses: docker/build-push-action/.github/workflows/reusable-distribute-mp.yml@v6
124+
with:
125+
push: ${{ github.event_name != 'pull_request' }}
126+
cache: true
127+
meta-image: user/app
128+
build-platforms: linux/amd64,linux/arm64
129+
login-username: ${{ vars.DOCKERHUB_USERNAME }}
130+
secrets:
131+
login-password: ${{ secrets.DOCKERHUB_TOKEN }}
226132
```
227133

228-
### With Bake
134+
Here are the main inputs for this reusable workflow:
135+
136+
| Name | Type | Default | Description |
137+
|-------------------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
138+
| `runner` | String | `auto`¹ | Runner instance (e.g., `ubuntu-latest`). |
139+
| `push` | Bool | `false` | Push image to registry. |
140+
| `cache` | Bool | `false` | Enable GitHub Actions cache backend. |
141+
| `cache-scope` | String | | Which scope GitHub Actions cache object belongs to if `cache` enabled. |
142+
| `cache-mode` | String | `min` | Cache layers to export if `cache` enabled (one of `min` or `max`). |
143+
| `meta-image` | String | | Image to use as base name for tags. This input is similar to [`images` input in `docker/metadata-action`](https://github.com/docker/metadata-action?tab=readme-ov-file#images-input) used in this reusable workflow but accepts a single image name. |
144+
| `build-platforms` | List/CSV | | List of target platforms for build. This input is similar to [`platforms` input in `docker/build-push-action`](https://github.com/docker/build-push-action?tab=readme-ov-file#inputs) used in this reusable workflow. At least two platforms are required. |
145+
| `login-username`² | String | | Username used to log against the Docker registry. This input is similar to [`username` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
146+
| `login-password` | String | | Specifies whether the given registry is ECR (auto, true or false). This input is similar to [`password` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
229147

230-
It's also possible to build on multiple runners using Bake, with the
231-
[bake action](https://github.com/docker/bake-action).
148+
> [!NOTE]
149+
> ¹ `auto` will choose the best matching runner depending on the target
150+
> platform being built (either `ubuntu-latest` or `ubuntu-24.04-arm`).
151+
>
152+
> ² `login-username` can be used as either an input or secret.
232153

233-
You can find a live example [in this GitHub repository](https://github.com/crazy-max/docker-linguist).
154+
You can find the full list of inputs in [the reusable workflow](https://github.com/docker/build-push-action/blob/v6/.github/workflows/reusable-distribute-mp.yml).
234155

235-
The following example achieves the same results as described in
236-
[the previous section](#distribute-build-across-multiple-runners).
156+
{{< /tab >}}
157+
{{< tab name="bake-action" >}}
237158

238159
```hcl
239160
variable "DEFAULT_TAG" {
@@ -270,144 +191,47 @@ target "image-all" {
270191
}
271192
```
272193

273-
```yaml
194+
```yaml {hl_lines=9}
274195
name: ci
275196
276197
on:
277198
push:
278-
279-
env:
280-
REGISTRY_IMAGE: user/app
199+
pull_request:
281200
282201
jobs:
283-
prepare:
284-
runs-on: ubuntu-latest
285-
outputs:
286-
matrix: ${{ steps.platforms.outputs.matrix }}
287-
steps:
288-
- name: Checkout
289-
uses: actions/checkout@v4
290-
291-
- name: Create matrix
292-
id: platforms
293-
run: |
294-
echo "matrix=$(docker buildx bake image-all --print | jq -cr '.target."image-all".platforms')" >>${GITHUB_OUTPUT}
295-
296-
- name: Show matrix
297-
run: |
298-
echo ${{ steps.platforms.outputs.matrix }}
299-
300-
- name: Docker meta
301-
id: meta
302-
uses: docker/metadata-action@v5
303-
with:
304-
images: ${{ env.REGISTRY_IMAGE }}
305-
306-
- name: Rename meta bake definition file
307-
run: |
308-
mv "${{ steps.meta.outputs.bake-file }}" "${{ runner.temp }}/bake-meta.json"
309-
310-
- name: Upload meta bake definition
311-
uses: actions/upload-artifact@v4
312-
with:
313-
name: bake-meta
314-
path: ${{ runner.temp }}/bake-meta.json
315-
if-no-files-found: error
316-
retention-days: 1
317-
318202
build:
319-
runs-on: ubuntu-latest
320-
needs:
321-
- prepare
322-
strategy:
323-
fail-fast: false
324-
matrix:
325-
platform: ${{ fromJson(needs.prepare.outputs.matrix) }}
326-
steps:
327-
- name: Prepare
328-
run: |
329-
platform=${{ matrix.platform }}
330-
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
331-
332-
- name: Download meta bake definition
333-
uses: actions/download-artifact@v4
334-
with:
335-
name: bake-meta
336-
path: ${{ runner.temp }}
337-
338-
- name: Login to Docker Hub
339-
uses: docker/login-action@v3
340-
with:
341-
username: ${{ vars.DOCKERHUB_USERNAME }}
342-
password: ${{ secrets.DOCKERHUB_TOKEN }}
343-
344-
- name: Set up QEMU
345-
uses: docker/setup-qemu-action@v3
346-
347-
- name: Set up Docker Buildx
348-
uses: docker/setup-buildx-action@v3
349-
350-
- name: Build
351-
id: bake
352-
uses: docker/bake-action@v6
353-
with:
354-
files: |
355-
./docker-bake.hcl
356-
cwd://${{ runner.temp }}/bake-meta.json
357-
targets: image
358-
set: |
359-
*.tags=${{ env.REGISTRY_IMAGE }}
360-
*.platform=${{ matrix.platform }}
361-
*.output=type=image,push-by-digest=true,name-canonical=true,push=true
362-
363-
- name: Export digest
364-
run: |
365-
mkdir -p ${{ runner.temp }}/digests
366-
digest="${{ fromJSON(steps.bake.outputs.metadata).image['containerimage.digest'] }}"
367-
touch "${{ runner.temp }}/digests/${digest#sha256:}"
368-
369-
- name: Upload digest
370-
uses: actions/upload-artifact@v4
371-
with:
372-
name: digests-${{ env.PLATFORM_PAIR }}
373-
path: ${{ runner.temp }}/digests/*
374-
if-no-files-found: error
375-
retention-days: 1
376-
377-
merge:
378-
runs-on: ubuntu-latest
379-
needs:
380-
- build
381-
steps:
382-
- name: Download meta bake definition
383-
uses: actions/download-artifact@v4
384-
with:
385-
name: bake-meta
386-
path: ${{ runner.temp }}
203+
uses: docker/bake-action/.github/workflows/reusable-distribute-mp.yml@v6
204+
with:
205+
target: image-all
206+
push: ${{ github.event_name != 'pull_request' }}
207+
cache: true
208+
meta-image: user/app
209+
login-username: ${{ vars.DOCKERHUB_USERNAME }}
210+
secrets:
211+
login-password: ${{ secrets.DOCKERHUB_TOKEN }}
212+
```
387213

388-
- name: Download digests
389-
uses: actions/download-artifact@v4
390-
with:
391-
path: ${{ runner.temp }}/digests
392-
pattern: digests-*
393-
merge-multiple: true
214+
Here are the main inputs for this reusable workflow:
394215

395-
- name: Login to DockerHub
396-
uses: docker/login-action@v3
397-
with:
398-
username: ${{ vars.DOCKERHUB_USERNAME }}
399-
password: ${{ secrets.DOCKERHUB_TOKEN }}
216+
| Name | Type | Default | Description |
217+
|-------------------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
218+
| `runner` | String | `auto`¹ | Runner instance (e.g., `ubuntu-latest`). |
219+
| `target` | String | | Multi-platform target to build. This input is similar to [`targets` input in `docker/bake-action`](https://github.com/docker/build-push-action?tab=readme-ov-file#inputs) used in this reusable workflow but accepts a single target. |
220+
| `push` | Bool | `false` | Push image to registry. |
221+
| `cache` | Bool | `false` | Enable GitHub Actions cache backend. |
222+
| `cache-scope` | String | | Which scope GitHub Actions cache object belongs to if `cache` enabled. |
223+
| `cache-mode` | String | `min` | Cache layers to export if `cache` enabled (one of `min` or `max`). |
224+
| `meta-image` | String | | Image to use as base name for tags. This input is similar to [`images` input in `docker/metadata-action`](https://github.com/docker/metadata-action?tab=readme-ov-file#images-input) used in this reusable workflow but accepts a single image name. |
225+
| `login-username`² | String | | Username used to log against the Docker registry. This input is similar to [`username` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
226+
| `login-password` | String | | Specifies whether the given registry is ECR (auto, true or false). This input is similar to [`password` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
400227

401-
- name: Set up Docker Buildx
402-
uses: docker/setup-buildx-action@v3
228+
> [!NOTE]
229+
> ¹ `auto` will choose the best matching runner depending on the target
230+
> platform being built (either `ubuntu-latest` or `ubuntu-24.04-arm`).
231+
>
232+
> ² `login-username` can be used as either an input or secret.
403233

404-
- name: Create manifest list and push
405-
working-directory: ${{ runner.temp }}/digests
406-
run: |
407-
docker buildx imagetools create $(jq -cr '.target."docker-metadata-action".tags | map(select(startswith("${{ env.REGISTRY_IMAGE }}")) | "-t " + .) | join(" ")' ${{ runner.temp }}/bake-meta.json) \
408-
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
234+
You can find the list of available inputs in [the reusable workflow](https://github.com/docker/bake-action/blob/v6/.github/workflows/reusable-distribute-mp.yml).
409235

410-
- name: Inspect image
411-
run: |
412-
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:$(jq -r '.target."docker-metadata-action".args.DOCKER_META_VERSION' ${{ runner.temp }}/bake-meta.json)
413-
```
236+
{{< /tab >}}
237+
{{< /tabs >}}

0 commit comments

Comments
 (0)