Skip to content

Commit 66e3181

Browse files
crazy-maxcrazy-max
committed
build: reusable workflows to distribute multi-platform build
Signed-off-by: CrazyMax <[email protected]>
1 parent 75edebe commit 66e3181

File tree

1 file changed

+76
-243
lines changed

1 file changed

+76
-243
lines changed

‎content/manuals/build/ci/github-actions/multi-platform.md‎

Lines changed: 76 additions & 243 deletions
Original file line numberDiff line numberDiff line change
@@ -98,137 +98,65 @@ jobs:
9898
tags: user/app:latest
9999
```
100100

101-
## Distribute build across multiple runners
101+
## Distribute multi-platform build across runners
102102

103103
In the previous example, each platform is built on the same runner which can
104104
take a long time depending on the number of platforms and your Dockerfile.
105105

106-
To solve this issue you can use a matrix strategy to distribute the build for
107-
each platform across multiple runners and create manifest list using the
108-
[`buildx imagetools create` command](/reference/cli/docker/buildx/imagetools/create.md).
106+
To solve this issue, you can use the following reusable workflows for both
107+
[`docker/build-push-action`](https://github.com/crazy-max/.github?tab=readme-ov-file#build-distribute-mp)
108+
and [`docker/bake-action`](https://github.com/crazy-max/.github?tab=readme-ov-file#bake-distribute-mp)
109+
actions to distribute multi-platform builds across runners efficiently.
109110

110-
The following workflow will build the image for each platform on a dedicated
111-
runner using a matrix strategy and push by digest. Then, the `merge` job will
112-
create manifest lists and push them to Docker Hub. The [`metadata` action](https://github.com/docker/metadata-action)
113-
is used to set tags and labels.
111+
{{< tabs >}}
112+
{{< tab name="build-push-action" >}}
114113

115-
```yaml
114+
```yaml {hl_lines=9}
116115
name: ci
117116
118117
on:
119118
push:
120-
121-
env:
122-
REGISTRY_IMAGE: user/app
119+
pull_request:
123120
124121
jobs:
125122
build:
126-
runs-on: ubuntu-latest
127-
strategy:
128-
fail-fast: false
129-
matrix:
130-
platform:
131-
- linux/amd64
132-
- linux/arm64
133-
steps:
134-
- name: Prepare
135-
run: |
136-
platform=${{ matrix.platform }}
137-
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
138-
139-
- name: Docker meta
140-
id: meta
141-
uses: docker/metadata-action@v5
142-
with:
143-
images: ${{ env.REGISTRY_IMAGE }}
144-
145-
- name: Login to Docker Hub
146-
uses: docker/login-action@v3
147-
with:
148-
username: ${{ vars.DOCKERHUB_USERNAME }}
149-
password: ${{ secrets.DOCKERHUB_TOKEN }}
150-
151-
- name: Set up QEMU
152-
uses: docker/setup-qemu-action@v3
153-
154-
- name: Set up Docker Buildx
155-
uses: docker/setup-buildx-action@v3
156-
157-
- name: Build and push by digest
158-
id: build
159-
uses: docker/build-push-action@v6
160-
with:
161-
platforms: ${{ matrix.platform }}
162-
labels: ${{ steps.meta.outputs.labels }}
163-
tags: ${{ env.REGISTRY_IMAGE }}
164-
outputs: type=image,push-by-digest=true,name-canonical=true,push=true
165-
166-
- name: Export digest
167-
run: |
168-
mkdir -p ${{ runner.temp }}/digests
169-
digest="${{ steps.build.outputs.digest }}"
170-
touch "${{ runner.temp }}/digests/${digest#sha256:}"
171-
172-
- name: Upload digest
173-
uses: actions/upload-artifact@v4
174-
with:
175-
name: digests-${{ env.PLATFORM_PAIR }}
176-
path: ${{ runner.temp }}/digests/*
177-
if-no-files-found: error
178-
retention-days: 1
179-
180-
merge:
181-
runs-on: ubuntu-latest
182-
needs:
183-
- build
184-
steps:
185-
- name: Download digests
186-
uses: actions/download-artifact@v4
187-
with:
188-
path: ${{ runner.temp }}/digests
189-
pattern: digests-*
190-
merge-multiple: true
191-
192-
- name: Login to Docker Hub
193-
uses: docker/login-action@v3
194-
with:
195-
username: ${{ vars.DOCKERHUB_USERNAME }}
196-
password: ${{ secrets.DOCKERHUB_TOKEN }}
197-
198-
- name: Set up Docker Buildx
199-
uses: docker/setup-buildx-action@v3
200-
201-
- name: Docker meta
202-
id: meta
203-
uses: docker/metadata-action@v5
204-
with:
205-
images: ${{ env.REGISTRY_IMAGE }}
206-
tags: |
207-
type=ref,event=branch
208-
type=ref,event=pr
209-
type=semver,pattern={{version}}
210-
type=semver,pattern={{major}}.{{minor}}
211-
212-
- name: Create manifest list and push
213-
working-directory: ${{ runner.temp }}/digests
214-
run: |
215-
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
216-
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
217-
218-
- name: Inspect image
219-
run: |
220-
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}
123+
uses: crazy-max/.github/.github/workflows/build-distribute-mp.yml@main
124+
with:
125+
push: ${{ github.event_name != 'pull_request' }}
126+
cache: true
127+
meta-image: user/app
128+
build-platforms: linux/amd64,linux/arm64
129+
login-username: ${{ vars.DOCKERHUB_USERNAME }}
130+
secrets:
131+
login-password: ${{ secrets.DOCKERHUB_TOKEN }}
221132
```
222133

223-
### With Bake
134+
Here are the main inputs for this reusable workflow:
135+
136+
| Name | Type | Default | Description |
137+
|-------------------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
138+
| `runner` | String | `auto`¹ | Runner instance (e.g., `ubuntu-latest`). |
139+
| `push` | Bool | `false` | Push image to registry. |
140+
| `cache` | Bool | `false` | Enable GitHub Actions cache backend. |
141+
| `cache-scope` | String | | Which scope GitHub Actions cache object belongs to if `cache` enabled. |
142+
| `cache-mode` | String | `min` | Cache layers to export if `cache` enabled (one of `min` or `max`). |
143+
| `summary` | Bool | `true` | Enable [build summary](https://docs.docker.com/build/ci/github-actions/build-summary/) generation. |
144+
| `meta-image` | String | | Image to use as base name for tags. This input is similar to [`images` input in `docker/metadata-action`](https://github.com/docker/metadata-action?tab=readme-ov-file#images-input) used in this reusable workflow but accepts a single image name. |
145+
| `build-platforms` | List/CSV | | List of target platforms for build. This input is similar to [`platforms` input in `docker/build-push-action`](https://github.com/docker/build-push-action?tab=readme-ov-file#inputs) used in this reusable workflow. At least two platforms are required. |
146+
| `login-registry` | String | | Server address of Docker registry. If not set then will default to Docker Hub. This input is similar to [`registry` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
147+
| `login-username`² | String | | Username used to log against the Docker registry. This input is similar to [`username` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
148+
| `login-password` | String | | Specifies whether the given registry is ECR (auto, true or false). This input is similar to [`password` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
224149

225-
It's also possible to build on multiple runners using Bake, with the
226-
[bake action](https://github.com/docker/bake-action).
150+
> [!NOTE]
151+
> ¹ `auto` will choose the best matching runner depending on the target
152+
> platform being built (either `ubuntu-latest` or `ubuntu-24.04-arm`).
153+
>
154+
> ² `login-username` can be used as either an input or secret.
227155

228-
You can find a live example [in this GitHub repository](https://github.com/crazy-max/docker-linguist).
156+
You can find the list of available inputs directly in [the reusable workflow](https://github.com/crazy-max/.github/blob/main/.github/workflows/build-distribute-mp.yml).
229157

230-
The following example achieves the same results as described in
231-
[the previous section](#distribute-build-across-multiple-runners).
158+
{{< /tab >}}
159+
{{< tab name="bake-action" >}}
232160

233161
```hcl
234162
variable "DEFAULT_TAG" {
@@ -265,144 +193,49 @@ target "image-all" {
265193
}
266194
```
267195

268-
```yaml
196+
```yaml {hl_lines=9}
269197
name: ci
270198
271199
on:
272200
push:
273-
274-
env:
275-
REGISTRY_IMAGE: user/app
201+
pull_request:
276202
277203
jobs:
278-
prepare:
279-
runs-on: ubuntu-latest
280-
outputs:
281-
matrix: ${{ steps.platforms.outputs.matrix }}
282-
steps:
283-
- name: Checkout
284-
uses: actions/checkout@v4
285-
286-
- name: Create matrix
287-
id: platforms
288-
run: |
289-
echo "matrix=$(docker buildx bake image-all --print | jq -cr '.target."image-all".platforms')" >>${GITHUB_OUTPUT}
290-
291-
- name: Show matrix
292-
run: |
293-
echo ${{ steps.platforms.outputs.matrix }}
294-
295-
- name: Docker meta
296-
id: meta
297-
uses: docker/metadata-action@v5
298-
with:
299-
images: ${{ env.REGISTRY_IMAGE }}
300-
301-
- name: Rename meta bake definition file
302-
run: |
303-
mv "${{ steps.meta.outputs.bake-file }}" "${{ runner.temp }}/bake-meta.json"
304-
305-
- name: Upload meta bake definition
306-
uses: actions/upload-artifact@v4
307-
with:
308-
name: bake-meta
309-
path: ${{ runner.temp }}/bake-meta.json
310-
if-no-files-found: error
311-
retention-days: 1
312-
313204
build:
314-
runs-on: ubuntu-latest
315-
needs:
316-
- prepare
317-
strategy:
318-
fail-fast: false
319-
matrix:
320-
platform: ${{ fromJson(needs.prepare.outputs.matrix) }}
321-
steps:
322-
- name: Prepare
323-
run: |
324-
platform=${{ matrix.platform }}
325-
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
326-
327-
- name: Download meta bake definition
328-
uses: actions/download-artifact@v4
329-
with:
330-
name: bake-meta
331-
path: ${{ runner.temp }}
332-
333-
- name: Login to Docker Hub
334-
uses: docker/login-action@v3
335-
with:
336-
username: ${{ vars.DOCKERHUB_USERNAME }}
337-
password: ${{ secrets.DOCKERHUB_TOKEN }}
338-
339-
- name: Set up QEMU
340-
uses: docker/setup-qemu-action@v3
341-
342-
- name: Set up Docker Buildx
343-
uses: docker/setup-buildx-action@v3
344-
345-
- name: Build
346-
id: bake
347-
uses: docker/bake-action@v6
348-
with:
349-
files: |
350-
./docker-bake.hcl
351-
cwd://${{ runner.temp }}/bake-meta.json
352-
targets: image
353-
set: |
354-
*.tags=${{ env.REGISTRY_IMAGE }}
355-
*.platform=${{ matrix.platform }}
356-
*.output=type=image,push-by-digest=true,name-canonical=true,push=true
357-
358-
- name: Export digest
359-
run: |
360-
mkdir -p ${{ runner.temp }}/digests
361-
digest="${{ fromJSON(steps.bake.outputs.metadata).image['containerimage.digest'] }}"
362-
touch "${{ runner.temp }}/digests/${digest#sha256:}"
363-
364-
- name: Upload digest
365-
uses: actions/upload-artifact@v4
366-
with:
367-
name: digests-${{ env.PLATFORM_PAIR }}
368-
path: ${{ runner.temp }}/digests/*
369-
if-no-files-found: error
370-
retention-days: 1
371-
372-
merge:
373-
runs-on: ubuntu-latest
374-
needs:
375-
- build
376-
steps:
377-
- name: Download meta bake definition
378-
uses: actions/download-artifact@v4
379-
with:
380-
name: bake-meta
381-
path: ${{ runner.temp }}
382-
383-
- name: Download digests
384-
uses: actions/download-artifact@v4
385-
with:
386-
path: ${{ runner.temp }}/digests
387-
pattern: digests-*
388-
merge-multiple: true
205+
uses: crazy-max/.github/.github/workflows/build-distribute-mp.yml@main
206+
with:
207+
target: image-all
208+
push: ${{ github.event_name != 'pull_request' }}
209+
cache: true
210+
meta-image: user/app
211+
login-username: ${{ vars.DOCKERHUB_USERNAME }}
212+
secrets:
213+
login-password: ${{ secrets.DOCKERHUB_TOKEN }}
214+
```
389215

390-
- name: Login to DockerHub
391-
uses: docker/login-action@v3
392-
with:
393-
username: ${{ vars.DOCKERHUB_USERNAME }}
394-
password: ${{ secrets.DOCKERHUB_TOKEN }}
216+
Here are the main inputs for this reusable workflow:
217+
218+
| Name | Type | Default | Description |
219+
|-------------------|--------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
220+
| `runner` | String | `auto`¹ | Runner instance (e.g., `ubuntu-latest`). |
221+
| `target` | String | | Multi-platform target to build. This input is similar to [`targets` input in `docker/bake-action`](https://github.com/docker/build-push-action?tab=readme-ov-file#inputs) used in this reusable workflow but accepts a single target. |
222+
| `push` | Bool | `false` | Push image to registry. |
223+
| `cache` | Bool | `false` | Enable GitHub Actions cache backend. |
224+
| `cache-scope` | String | | Which scope GitHub Actions cache object belongs to if `cache` enabled. |
225+
| `cache-mode` | String | `min` | Cache layers to export if `cache` enabled (one of `min` or `max`). |
226+
| `summary` | Bool | `true` | Enable [build summary](https://docs.docker.com/build/ci/github-actions/build-summary/) generation. |
227+
| `meta-image` | String | | Image to use as base name for tags. This input is similar to [`images` input in `docker/metadata-action`](https://github.com/docker/metadata-action?tab=readme-ov-file#images-input) used in this reusable workflow but accepts a single image name. |
228+
| `login-registry` | String | | Server address of Docker registry. If not set then will default to Docker Hub. This input is similar to [`registry` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
229+
| `login-username`² | String | | Username used to log against the Docker registry. This input is similar to [`username` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
230+
| `login-password` | String | | Specifies whether the given registry is ECR (auto, true or false). This input is similar to [`password` input in `docker/login-action`](https://github.com/docker/login-action?tab=readme-ov-file#inputs) used in this reusable workflow. |
395231

396-
- name: Set up Docker Buildx
397-
uses: docker/setup-buildx-action@v3
232+
> [!NOTE]
233+
> ¹ `auto` will choose the best matching runner depending on the target
234+
> platform being built (either `ubuntu-latest` or `ubuntu-24.04-arm`).
235+
>
236+
> ² `login-username` can be used as either an input or secret.
398237

399-
- name: Create manifest list and push
400-
working-directory: ${{ runner.temp }}/digests
401-
run: |
402-
docker buildx imagetools create $(jq -cr '.target."docker-metadata-action".tags | map(select(startswith("${{ env.REGISTRY_IMAGE }}")) | "-t " + .) | join(" ")' ${{ runner.temp }}/bake-meta.json) \
403-
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
238+
You can find the list of available inputs directly in [the reusable workflow](https://github.com/crazy-max/.github/blob/main/.github/workflows/bake-distribute-mp.yml).
404239

405-
- name: Inspect image
406-
run: |
407-
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:$(jq -r '.target."docker-metadata-action".args.DOCKER_META_VERSION' ${{ runner.temp }}/bake-meta.json)
408-
```
240+
{{< /tab >}}
241+
{{< /tabs >}}

0 commit comments

Comments
 (0)