Skip to content

Commit 16c4723

Browse files
thirtytwobitsthirtytwobits
authored
Toxic/mp (#41)
* Improving build performance and upgrading doxygen This change started as a simple upgrade to doxygen in the toolshed but build times were so bad I ended up removing arm-none-eabi which we weren't using anyway. * Adding GCC-12. closes #40
1 parent aed5fcb commit 16c4723

17 files changed

+576
-225
lines changed

‎.github/workflows/toolshed.yml‎

Lines changed: 82 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -8,37 +8,31 @@ on:
88
branches: [ "main" ]
99

1010
env:
11-
ORG_NAMESPACE: opencyphal
12-
IMAGE_NAME: toolshed
11+
REGISTRY_IMAGE: ghcr.io/opencyphal/toolshed
1312
IMAGE_VERSION_PREFIX: ts
1413

1514
jobs:
16-
17-
dry_run:
15+
release-ts:
1816
if: ${{ github.event_name == 'pull_request' || ( github.event_name == 'release' && startsWith( github.ref, 'refs/tags/ts') && github.event.release.prerelease ) }}
1917
runs-on: ubuntu-latest
20-
18+
strategy:
19+
matrix:
20+
platform:
21+
- linux/amd64
22+
- linux/arm64
2123
steps:
22-
- uses: actions/checkout@v3
23-
24-
- # Add QEMU to allow building non-native containers as
25-
# part of multi-platform container builds.
26-
name: Set up QEMU
27-
uses: docker/setup-qemu-action@v3
28-
29-
- # Setup buildx so we can do multi-platform container builds.
30-
name: Set up Docker Buildx
31-
uses: docker/setup-buildx-action@v3
24+
- name: Prepare
25+
run: |
26+
platform=${{ matrix.platform }}
27+
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
3228
33-
- name: build
34-
run: cd ${{ env.IMAGE_NAME}} && docker buildx build --platform linux/arm64,linux/amd64 .
29+
- uses: actions/checkout@v4
3530

36-
release:
37-
if: ${{ github.event_name == 'release' && startsWith( github.ref, 'refs/tags/ts' ) && !github.event.release.prerelease }}
38-
runs-on: ubuntu-latest
39-
40-
steps:
41-
- uses: actions/checkout@v3
31+
- name: Docker meta
32+
id: meta
33+
uses: docker/metadata-action@v5
34+
with:
35+
images: ${{ env.REGISTRY_IMAGE }}
4236

4337
- # Add QEMU to allow building non-native containers as
4438
# part of multi-platform container builds.
@@ -49,16 +43,70 @@ jobs:
4943
name: Set up Docker Buildx
5044
uses: docker/setup-buildx-action@v3
5145

52-
- name: Login to GHCR
53-
uses: docker/login-action@v2
46+
- name: Build and push by digest
47+
id: build
48+
uses: docker/build-push-action@v5
5449
with:
55-
registry: ghcr.io
56-
username: ${{ github.repository_owner }}
57-
password: ${{ secrets.GITHUB_TOKEN }}
58-
59-
- name: form image tag
60-
id: image_tag
61-
uses: olegtarasov/[email protected]
50+
context: toolshed
51+
platforms: ${{ matrix.platform }}
52+
labels: ${{ steps.meta.outputs.labels }}
53+
outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=${{ github.event_name == 'pull_request' && 'false' || 'true' }}
54+
-
55+
name: Export digest
56+
if: ${{ github.event_name != 'pull_request'}}
57+
run: |
58+
mkdir -p /tmp/digests
59+
digest="${{ steps.build.outputs.digest }}"
60+
touch "/tmp/digests/${digest#sha256:}"
61+
-
62+
name: Upload digest
63+
if: ${{ github.event_name != 'pull_request'}}
64+
uses: actions/upload-artifact@v4
65+
with:
66+
name: digests-${{ env.PLATFORM_PAIR }}
67+
path: /tmp/digests/*
68+
if-no-files-found: error
69+
retention-days: 1
6270

63-
- name: build-and-push
64-
run: cd ${{ env.IMAGE_NAME }} && docker buildx build --platform linux/arm64,linux/amd64 --push -t ghcr.io/${{ env.ORG_NAMESPACE }}/${{ env.IMAGE_NAME }}:${{ steps.image_tag.outputs.tag }} .
71+
merge-ts:
72+
if: ${{ github.event_name == 'release' && startsWith( github.ref, 'refs/tags/ts' ) && !github.event.release.prerelease }}
73+
runs-on: ubuntu-latest
74+
needs:
75+
- release-ts
76+
steps:
77+
-
78+
name: Download digests
79+
uses: actions/download-artifact@v4
80+
with:
81+
path: /tmp/digests
82+
pattern: digests-*
83+
merge-multiple: true
84+
-
85+
name: Set up Docker Buildx
86+
uses: docker/setup-buildx-action@v3
87+
88+
-
89+
name: Docker meta
90+
id: meta
91+
uses: docker/metadata-action@v5
92+
with:
93+
tags: type=ref,event=tag
94+
images: ${{ env.REGISTRY_IMAGE }}
95+
96+
- name: Login to GHCR
97+
uses: docker/login-action@v2
98+
with:
99+
registry: ghcr.io
100+
username: ${{ github.repository_owner }}
101+
password: ${{ secrets.GITHUB_TOKEN }}
102+
103+
-
104+
name: Create manifest list and push
105+
working-directory: /tmp/digests
106+
run: |
107+
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
108+
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
109+
-
110+
name: Inspect image
111+
run: |
112+
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

‎.github/workflows/toxic.yml‎

Lines changed: 82 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -8,37 +8,31 @@ on:
88
branches: [ "main" ]
99

1010
env:
11-
ORG_NAMESPACE: opencyphal
12-
IMAGE_NAME: toxic
11+
REGISTRY_IMAGE: ghcr.io/opencyphal/toxic
1312
IMAGE_VERSION_PREFIX: tx
1413

1514
jobs:
16-
17-
dry_run:
15+
release-tx:
1816
if: ${{ github.event_name == 'pull_request' || ( github.event_name == 'release' && startsWith( github.ref, 'refs/tags/tx') && github.event.release.prerelease ) }}
1917
runs-on: ubuntu-latest
20-
18+
strategy:
19+
matrix:
20+
platform:
21+
- linux/amd64
22+
- linux/arm64
2123
steps:
22-
- uses: actions/checkout@v3
23-
24-
- # Add QEMU to allow building non-native containers as
25-
# part of multi-platform container builds.
26-
name: Set up QEMU
27-
uses: docker/setup-qemu-action@v3
28-
29-
- # Setup buildx so we can do multi-platform container builds.
30-
name: Set up Docker Buildx
31-
uses: docker/setup-buildx-action@v3
24+
- name: Prepare
25+
run: |
26+
platform=${{ matrix.platform }}
27+
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
3228
33-
- name: build
34-
run: cd ${{ env.IMAGE_NAME }} && docker buildx build --platform linux/arm64,linux/amd64 .
29+
- uses: actions/checkout@v4
3530

36-
release:
37-
if: ${{ github.event_name == 'release' && startsWith( github.ref, 'refs/tags/tx' ) && !github.event.release.prerelease }}
38-
runs-on: ubuntu-latest
39-
40-
steps:
41-
- uses: actions/checkout@v3
31+
- name: Docker meta
32+
id: meta
33+
uses: docker/metadata-action@v5
34+
with:
35+
images: ${{ env.REGISTRY_IMAGE }}
4236

4337
- # Add QEMU to allow building non-native containers as
4438
# part of multi-platform container builds.
@@ -49,16 +43,70 @@ jobs:
4943
name: Set up Docker Buildx
5044
uses: docker/setup-buildx-action@v3
5145

52-
- name: Login to GHCR
53-
uses: docker/login-action@v2
46+
- name: Build and push by digest
47+
id: build
48+
uses: docker/build-push-action@v5
5449
with:
55-
registry: ghcr.io
56-
username: ${{ github.repository_owner }}
57-
password: ${{ secrets.GITHUB_TOKEN }}
58-
59-
- name: form image tag
60-
id: image_tag
61-
uses: olegtarasov/[email protected]
50+
context: toxic
51+
platforms: ${{ matrix.platform }}
52+
labels: ${{ steps.meta.outputs.labels }}
53+
outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=${{ github.event_name == 'pull_request' && 'false' || 'true' }}
54+
-
55+
name: Export digest
56+
if: ${{ github.event_name != 'pull_request'}}
57+
run: |
58+
mkdir -p /tmp/digests
59+
digest="${{ steps.build.outputs.digest }}"
60+
touch "/tmp/digests/${digest#sha256:}"
61+
-
62+
name: Upload digest
63+
if: ${{ github.event_name != 'pull_request'}}
64+
uses: actions/upload-artifact@v4
65+
with:
66+
name: digests-${{ env.PLATFORM_PAIR }}
67+
path: /tmp/digests/*
68+
if-no-files-found: error
69+
retention-days: 1
6270

63-
- name: build-and-push
64-
run: cd ${{ env.IMAGE_NAME }} && docker buildx build --platform linux/arm64,linux/amd64 --push -t ghcr.io/${{ env.ORG_NAMESPACE }}/${{ env.IMAGE_NAME }}:${{ steps.image_tag.outputs.tag }} .
71+
merge-tx:
72+
if: ${{ github.event_name == 'release' && startsWith( github.ref, 'refs/tags/tx' ) && !github.event.release.prerelease }}
73+
runs-on: ubuntu-latest
74+
needs:
75+
- release-tx
76+
steps:
77+
-
78+
name: Download digests
79+
uses: actions/download-artifact@v4
80+
with:
81+
path: /tmp/digests
82+
pattern: digests-*
83+
merge-multiple: true
84+
-
85+
name: Set up Docker Buildx
86+
uses: docker/setup-buildx-action@v3
87+
88+
-
89+
name: Docker meta
90+
id: meta
91+
uses: docker/metadata-action@v5
92+
with:
93+
tags: type=ref,event=tag
94+
images: ${{ env.REGISTRY_IMAGE }}
95+
96+
- name: Login to GHCR
97+
uses: docker/login-action@v2
98+
with:
99+
registry: ghcr.io
100+
username: ${{ github.repository_owner }}
101+
password: ${{ secrets.GITHUB_TOKEN }}
102+
103+
-
104+
name: Create manifest list and push
105+
working-directory: /tmp/digests
106+
run: |
107+
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
108+
$(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
109+
-
110+
name: Inspect image
111+
run: |
112+
docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

‎README.md‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,13 @@ like [libcanard](https://github.com/OpenCyphal/libcanard) and
2525

2626
| tag | Python | Gcc | Clang | Cmake | Platforms | Other Utilities |
2727
|----------|--------|-----|-------|-------|-----------|-----------------|
28-
| [ts20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 9.3.0 | 10.0.0 | 3.16.3 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li></ul> |
29-
| [ts22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li><li>doxygen 1.9.6</li></ul>
30-
| [ts22.4.2](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>~~sonar-scanner~~</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li></ul>
31-
| [ts22.4.3](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
32-
| ts22.4.4 - An experimental release that shouldn't be used. |
28+
| [ts22.4.7](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 12.3.0 | 18.1.3 | 3.22.1 | <ul><li>linux/amd64</li><li>linux/arm64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.10.0</li><li>nvm</li><li>node 20.x</li><li>nox</li><li>govr</li></ul>
29+
| [ts22.4.6](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.4.0 | 18.0.0 | 3.22.1 | <ul><li>linux/amd64</li><li>linux/arm64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.10.0</li><li>nvm</li><li>node 20.x</li><li>nox</li><li>govr</li></ul>
3330
| [ts22.4.5](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.4.0 | 18.0.0 | 3.22.1 | <ul><li>linux/amd64</li><li>linux/arm64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
31+
| [ts22.4.3](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
32+
| [ts22.4.2](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>~~sonar-scanner~~</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li></ul>
33+
| [ts22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li><li>doxygen 1.9.6</li></ul>
34+
3435

3536

3637
You can use this in your workflow yaml like this:
@@ -39,7 +40,7 @@ You can use this in your workflow yaml like this:
3940
jobs:
4041
my-job:
4142
runs-on: ubuntu-latest
42-
container: ghcr.io/opencyphal/toolshed:ts22.4.3
43+
container: ghcr.io/opencyphal/toolshed:ts22.4.7
4344
```
4445

4546

@@ -53,8 +54,8 @@ is based on Ubuntu and provides the necessary compilers and utilities to author
5354

5455
| tag | Python | Tex Live | git | Platforms | Other Utilities |
5556
|----------|--------|----------|-----|-----------|-----------------|
56-
| [te20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2019 | 2.25.1 | <ul><li>linux/amd64</li></ul> | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
5757
| [te22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2022 | 2.34.1 | <ul><li>linux/amd64</li></ul> | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
58+
| [te20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2019 | 2.25.1 | <ul><li>linux/amd64</li></ul> | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
5859

5960
You can use this in your workflow yaml like this:
6061

@@ -78,17 +79,18 @@ is based on Ubuntu and provides all modern Python3 distributions, pip, tox, and
7879

7980
| tag | Base Python | Python Versions | Tox | Nox | pip | Platforms | Other Utilities |
8081
|----------|-------------|-----------------|-----|-----|-----|-----------|-----------------|
81-
| [tx20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | <ul><li>linux/amd64</li></ul> | <ul><li>sonar-scanner</li></ul> |
82-
| [tx22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | 2023.4.22 | 20.0.2 | <ul><li>linux/amd64</li></ul> | |
8382
| [tx22.4.2](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.10 | 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.13.0 | 2023.4.22 | 20.0.2 | <ul><li>linux/amd64</li><li>linux/arm64</li></ul> | |
83+
| [tx22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | 2023.4.22 | 20.0.2 | <ul><li>linux/amd64</li></ul> | |
84+
| [tx20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | <ul><li>linux/amd64</li></ul> | <ul><li>sonar-scanner</li></ul> |
85+
8486

8587
You can use this in your workflow yaml like this:
8688

8789
```none
8890
jobs:
8991
my-job:
9092
runs-on: ubuntu-latest
91-
container: ghcr.io/opencyphal/toxic:tx20.4.1
93+
container: ghcr.io/opencyphal/toxic:tx20.4.2
9294
```
9395

9496
---------------------------

0 commit comments

Comments
 (0)