Skip to content

Commit 4f6af2d

Browse files
thirtytwobitsthirtytwobits
authored
Fix for #35 and first multi-platform build (#37)
This builds llvm 18 and builds for both amd64 and arm64 hosts.
1 parent 5d1c4bf commit 4f6af2d

12 files changed

+245
-103
lines changed

‎.github/workflows/texer.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: ghcr.io/opencyphal/texer container build and publish.
33
on:
44
release:
55
types: [ published ]
6-
6+
77
pull_request:
88
branches: [ "main" ]
99

‎.github/workflows/toolshed.yml‎

Lines changed: 37 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: ghcr.io/opencyphal/toolshed container build and publish.
33
on:
44
release:
55
types: [ published ]
6-
6+
77
pull_request:
88
branches: [ "main" ]
99

@@ -14,27 +14,47 @@ env:
1414

1515
jobs:
1616

17-
build_and_publish:
17+
dry_run:
18+
if: ${{ github.event_name == 'pull_request' || ( github.event_name == 'release' && startsWith( github.ref, 'ts') && github.event.release.prerelease ) }}
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- uses: actions/checkout@v3
23+
24+
- # Add QEMU to allow building non-native containers as
25+
# part of multi-platform container builds.
26+
name: Set up QEMU
27+
uses: docker/setup-qemu-action@v3
28+
29+
- # Setup buildx so we can do multi-platform container builds.
30+
name: Set up Docker Buildx
31+
uses: docker/setup-buildx-action@v3
1832

33+
- name: build
34+
run: cd ${{ env.IMAGE_NAME}} && docker buildx build --platform linux/arm64,linux/amd64 .
35+
36+
release:
37+
if: ${{ github.event_name == 'release' && startsWith( github.ref, 'ts' ) && !github.event.release.prerelease }}
1938
runs-on: ubuntu-latest
2039

2140
steps:
2241
- uses: actions/checkout@v3
2342

24-
- name: process event
25-
id: process_event
26-
uses: olegtarasov/[email protected]
43+
- # Add QEMU to allow building non-native containers as
44+
# part of multi-platform container builds.
45+
name: Set up QEMU
46+
uses: docker/setup-qemu-action@v3
47+
48+
- # Setup buildx so we can do multi-platform container builds.
49+
name: Set up Docker Buildx
50+
uses: docker/setup-buildx-action@v3
51+
52+
- name: Login to GHCR
53+
uses: docker/login-action@v2
2754
with:
28-
tagRegex: "(${{ env.IMAGE_VERSION_PREFIX }}).*"
29-
- name: dryrun-build
30-
if: ${{ github.event_name == 'pull_request' || ( github.event_name == 'release' && startsWith( steps.process_event.outputs.tag, env.IMAGE_VERSION_PREFIX ) && github.event.release.prerelease ) }}
31-
run: docker build ./${{ env.IMAGE_NAME }}
55+
registry: ghcr.io
56+
username: ${{ github.repository_owner }}
57+
password: ${{ secrets.GITHUB_TOKEN }}
58+
3259
- name: build-and-push
33-
if: ${{ github.event_name == 'release' && startsWith( steps.process_event.outputs.tag, env.IMAGE_VERSION_PREFIX ) && !github.event.release.prerelease }}
34-
uses: macbre/push-to-ghcr@master
35-
with:
36-
context: ./${{ env.IMAGE_NAME }}
37-
dockerfile: ./${{ env.IMAGE_NAME }}/Dockerfile
38-
image_name: ${{ env.ORG_NAMESPACE }}/${{ env.IMAGE_NAME }}
39-
github_token: ${{ secrets.GITHUB_TOKEN }}
40-
repository: ghcr.io
60+
run: cd ${{ env.IMAGE_NAME }} && docker buildx build --platform linux/arm64,linux/amd64 --push -t ghcr.io/${{ env.ORG_NAMESPACE }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_VERSION_PREFIX }}${{ github.ref }} .

‎.github/workflows/toxic.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: ghcr.io/opencyphal/toxic container build and publish.
33
on:
44
release:
55
types: [ published ]
6-
6+
77
pull_request:
88
branches: [ "main" ]
99

‎.vscode/settings.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"git-blame.gitWebUrl": ""
3+
}

‎README.md‎

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,14 @@ like [libcanard](https://github.com/OpenCyphal/libcanard) and
2323

2424
#### Supported Versions
2525

26-
| tag | Python | Gcc | Clang | Cmake | Other Utilities |
27-
|----------|--------|-----|-------|-------|-----------------|
28-
| [ts20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 9.3.0 | 10.0.0 | 3.16.3 | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li></ul> |
29-
| [ts22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li><li>doxygen 1.9.6</li></ul>
30-
| [ts22.4.2](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>qemu</li><li>can-utils</li><li>~~sonar-scanner~~</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li></ul>
31-
| [ts22.4.3](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
26+
| tag | Python | Gcc | Clang | Cmake | Platforms | Other Utilities |
27+
|----------|--------|-----|-------|-------|-----------|-----------------|
28+
| [ts20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 9.3.0 | 10.0.0 | 3.16.3 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li></ul> |
29+
| [ts22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>sonar-scanner</li><li>doxygen 1.9.6</li></ul>
30+
| [ts22.4.2](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>~~sonar-scanner~~</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li></ul>
31+
| [ts22.4.3](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.3.0 | 15.0.7 | 3.22.1 | <ul><li>linux/amd64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
32+
| ts22.4.4 - An experimental release that shouldn't be used. |
33+
| [ts22.4.5](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toolshed) | 3.10 | 11.4.0 | 18.0.0 | 3.22.1 | <ul><li>linux/amd64</li><li>linux/arm64</li></ul> | <ul><li>qemu</li><li>can-utils</li><li>doxygen 1.9.6</li><li>nvm</li><li>node 16.20.0</li><li>nox</li><li>govr</li></ul>
3234

3335

3436
You can use this in your workflow yaml like this:
@@ -49,10 +51,10 @@ is based on Ubuntu and provides the necessary compilers and utilities to author
4951

5052
#### Supported Versions
5153

52-
| tag | Python | Tex Live | git | Other Utilities |
53-
|----------|--------|----------|-----|-----------------|
54-
| [te20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2019 | 2.25.1 | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
55-
| [te22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2022 | 2.34.1 | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
54+
| tag | Python | Tex Live | git | Platforms | Other Utilities |
55+
|----------|--------|----------|-----|-----------|-----------------|
56+
| [te20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2019 | 2.25.1 | <ul><li>linux/amd64</li></ul> | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
57+
| [te22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/texer) | 3.11 | 2022 | 2.34.1 | <ul><li>linux/amd64</li></ul> | <ul><li>python pygments</li><li>lyx</li><li>inkscape</li></ul> |
5658

5759
You can use this in your workflow yaml like this:
5860

@@ -74,10 +76,10 @@ is based on Ubuntu and provides all modern Python3 distributions, pip, tox, and
7476

7577
#### Supported Versions
7678

77-
| tag | Base Python | Python Versions | Tox | Nox | pip | Other Utilities |
78-
|----------|-------------|-----------------|-----|-----|-----|-----------------|
79-
| [tx20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | <ul><li>sonar-scanner</li></ul> |
80-
| [tx22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | |
79+
| tag | Base Python | Python Versions | Tox | Nox | pip | Platforms | Other Utilities |
80+
|----------|-------------|-----------------|-----|-----|-----|-----------|-----------------|
81+
| [tx20.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | <ul><li>linux/amd64</li></ul> | <ul><li>sonar-scanner</li></ul> |
82+
| [tx22.4.1](https://github.com/OpenCyphal/docker_toolchains/pkgs/container/toxic) | 3.8 | 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 | 4.4.5 | (not available) | 20.0.2 | <ul><li>linux/amd64</li></ul> | |
8183

8284
You can use this in your workflow yaml like this:
8385

‎toolshed/Dockerfile‎

Lines changed: 69 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#
22
# Builds the toolshed/ubuntu-22.04 toolchain container.
33
#
4-
FROM ubuntu:22.04
4+
FROM ubuntu:22.04 AS base
55

66
LABEL org.opencontainers.image.description "Provides a consistent build and test environment for development, continuous-integration, and test automation of OpenCyphal C and C++ based projects."
77

@@ -19,53 +19,60 @@ ENV LC_ALL=en_US.UTF-8
1919
COPY provision.sh /
2020
RUN /provision.sh
2121

22+
2223
# +---------------------------------------------------------------------------+
2324
# | NVM/nodejs
2425
# +---------------------------------------------------------------------------+
26+
FROM base AS node
27+
2528
WORKDIR /tmp
2629
ADD nvm-install.sh ./nvm-install.sh
2730
RUN . ./nvm-install.sh
2831
ENV NVM_DIR="~/.nvm"
2932
RUN . ~/.nvm/nvm.sh && nvm install v16.20.0
3033

34+
35+
# +---------------------------------------------------------------------------+
36+
# | INSTALL PYTHON STUFF
37+
# +---------------------------------------------------------------------------+
38+
FROM node AS python
39+
WORKDIR /tmp
40+
41+
RUN pip3 install virtualenv
42+
RUN pip3 install tox
43+
RUN pip3 install nox
44+
RUN pip3 install gcovr
45+
RUN pip3 list
46+
47+
3148
# +---------------------------------------------------------------------------+
3249
# | LLVM
3350
# +---------------------------------------------------------------------------+
51+
FROM python AS llvm
3452
WORKDIR /tmp
53+
3554
ADD llvm-install.sh ./llvm-install.sh
3655
ADD llvm-select.sh ./llvm-select.sh
37-
RUN ./llvm-install.sh 15 all
38-
RUN ./llvm-select.sh 15
56+
RUN ./llvm-install.sh 18 all
57+
RUN ./llvm-select.sh 18
58+
3959

4060
# +---------------------------------------------------------------------------+
41-
# | DOXYGEN BUILD
61+
# | GCC
4262
# +---------------------------------------------------------------------------+
43-
ARG DOXYGEN_VERSION=1_9_6
44-
45-
WORKDIR /tmp
46-
COPY doxygen-Release_${DOXYGEN_VERSION}.sha512.txt .
47-
ADD https://github.com/doxygen/doxygen/archive/refs/tags/Release_${DOXYGEN_VERSION}.tar.gz doxygen-Release_${DOXYGEN_VERSION}.tar.gz
48-
RUN sha512sum -c doxygen-Release_${DOXYGEN_VERSION}.sha512.txt
49-
RUN mkdir doxygen-Release
50-
RUN tar -xvf doxygen-Release_${DOXYGEN_VERSION}.tar.gz -C doxygen-Release --strip-components 1
51-
RUN rm doxygen-Release_${DOXYGEN_VERSION}.tar.gz
52-
RUN mkdir doxygen-Release/build
63+
FROM llvm AS gcc-arm64
5364

54-
WORKDIR /tmp/doxygen-Release/build
55-
RUN cmake -G "Unix Makefiles" -Dbuild_parse:BOOL=ON -Duse_libclang:BOOL=ON ..
56-
RUN make
57-
RUN make install
65+
ARG GCC_HOST_PLATFORM=aarch64
5866

59-
WORKDIR /tmp
60-
RUN rm -rf doxygen-Release
67+
FROM llvm AS gcc-amd64
6168

69+
ARG GCC_HOST_PLATFORM=x86_64
6270

63-
# +---------------------------------------------------------------------------+
64-
# | INSTALL TOOLCHAINS
65-
# +---------------------------------------------------------------------------+
71+
FROM gcc-$TARGETARCH AS gcc
72+
WORKDIR /tmp
6673

6774
# ARM: M-Cores, Bare Metal
68-
ARG GCC_ARM_NONE_EABI_PLATFORM=x86_64
75+
ARG GCC_ARM_NONE_EABI_PLATFORM=${GCC_HOST_PLATFORM}
6976
ARG GCC_ARM_NONE_EABI_VERSION=11.2-2022.02
7077
ARG GCC_ARM_NONE_EABI_TRIPLE=arm-none-eabi
7178
ARG GCC_ARM_NONE_EABI_BASENAME=gcc-arm-${GCC_ARM_NONE_EABI_VERSION}-${GCC_ARM_NONE_EABI_PLATFORM}-${GCC_ARM_NONE_EABI_TRIPLE}
@@ -84,7 +91,7 @@ ENV PATH="/usr/share/${GCC_ARM_NONE_EABI_BASENAME}/bin:${PATH}"
8491
RUN ${GCC_ARM_NONE_EABI_TRIPLE}-gcc --version
8592

8693
# ARM: A-Cores, Linux
87-
ARG GCC_ARM_NONE_LINUX_PLATFORM=x86_64
94+
ARG GCC_ARM_NONE_LINUX_PLATFORM=${GCC_HOST_PLATFORM}
8895
ARG GCC_ARM_NONE_LINUX_VERSION=11.2-2022.02
8996
ARG GCC_ARM_NONE_LINUX_TRIPLE=arm-none-linux-gnueabihf
9097
ARG GCC_ARM_NONE_LINUX_BASENAME=gcc-arm-${GCC_ARM_NONE_LINUX_VERSION}-${GCC_ARM_NONE_LINUX_PLATFORM}-${GCC_ARM_NONE_LINUX_TRIPLE}
@@ -101,21 +108,47 @@ ENV PATH="/usr/share/${GCC_ARM_NONE_LINUX_BASENAME}/bin:${PATH}"
101108

102109
RUN ${GCC_ARM_NONE_LINUX_TRIPLE}-gcc --version
103110

104-
# +---------------------------------------------------------------------------+
105-
# | INSTALL THE PATH FOR INTERATIVE SESSIONS
106-
# +---------------------------------------------------------------------------+
107-
RUN echo "export PATH=$PATH" >> ~/.bashrc
108111

109112
# +---------------------------------------------------------------------------+
110-
# | INSTALL PYTHON STUFF
113+
# | DOXYGEN BUILD
111114
# +---------------------------------------------------------------------------+
112-
RUN pip3 install virtualenv
113-
RUN pip3 install tox
114-
RUN pip3 install nox
115-
RUN pip3 install gcovr
116-
RUN pip3 list
115+
FROM gcc AS doxygen-arm64
116+
117+
ARG USE_LIB_CLANG=OFF
118+
119+
FROM gcc AS doxygen-amd64
120+
121+
ARG USE_LIB_CLANG=OFF
122+
123+
FROM doxygen-$TARGETARCH AS doxygen
124+
WORKDIR /tmp
125+
126+
ARG DOXYGEN_VERSION=1_9_6
127+
128+
COPY doxygen-Release_${DOXYGEN_VERSION}.sha512.txt .
129+
ADD https://github.com/doxygen/doxygen/archive/refs/tags/Release_${DOXYGEN_VERSION}.tar.gz doxygen-Release_${DOXYGEN_VERSION}.tar.gz
130+
RUN sha512sum -c doxygen-Release_${DOXYGEN_VERSION}.sha512.txt
131+
RUN mkdir doxygen-Release
132+
RUN tar -xvf doxygen-Release_${DOXYGEN_VERSION}.tar.gz -C doxygen-Release --strip-components 1
133+
RUN rm doxygen-Release_${DOXYGEN_VERSION}.tar.gz
134+
RUN mkdir doxygen-Release/build
135+
136+
WORKDIR /tmp/doxygen-Release/build
137+
RUN cmake -G "Unix Makefiles" -Dbuild_parse:BOOL=ON -Duse_libclang:BOOL=${USE_LIB_CLANG} ..
138+
RUN make
139+
RUN make install
140+
141+
WORKDIR /tmp
142+
RUN rm -rf doxygen-Release
143+
117144

118145
# +---------------------------------------------------------------------------+
119-
# | LEAVE THE WORKDIR AS /repo
146+
# | FINALIZE
120147
# +---------------------------------------------------------------------------+
148+
FROM doxygen AS all
149+
150+
# INSTALL THE PATH FOR INTERATIVE SESSIONS
151+
RUN echo "export PATH=$PATH" >> ~/.bashrc
152+
153+
# LEAVE THE WORKDIR AS /repo
121154
WORKDIR /repo

‎toolshed/README.md‎

Lines changed: 53 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,22 +29,69 @@ export FGP = (fine-grained permission for OpenCyphal organization)
2929
echo $FGP | docker login ghcr.io -u (github username) --password-stdin
3030
```
3131

32-
... now build (where x is the next version number for the container):
32+
... build a multi-platform image following the instructions [here](https://docs.docker.com/build/building/multi-platform/#multiple-native-nodes):
3333

3434
```bash
35-
docker build -t ghcr.io/opencyphal/toolshed:ts22.4.x .
35+
docker buildx create --use --name cyphalbuild
3636
```
3737

38-
... and finally, push.
38+
If you already created the `cyphalbuild` builder then just use it instead of creating it:
3939

4040
```bash
41-
docker push ghcr.io/opencyphal/toolshed:ts22.4.x
41+
docker buildx use cyphalbuild
4242
```
4343

44-
## Testing out the container
44+
... then build the container:
4545

46-
To login to an interactive session do:
46+
```bash
47+
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/opencyphal/toolshed:ts22.4.x .
48+
```
49+
50+
(where x is the next version number for the container)
51+
52+
When the build completes you'll see the following warning:
53+
54+
> WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
55+
56+
It's important you don't restart your docker build container before you load and/or push since it may blow away the cache you need to push or load from.
57+
58+
### Load
59+
Currently you can't use --load with docker desktop so the only way to test the container is to rebuild for one platform only:
60+
61+
```bash
62+
docker buildx build --platform linux/arm64 --load -t ghcr.io/opencyphal/toolshed:ts22.4.x .
63+
```
64+
65+
This will be operating off of the cache so it shouldn't take very long to complete. After it does you'll see your image using the classic `docker images` command. Now you can login to the container to test it out:
4766

4867
```bash
4968
docker run --rm -it -v ${PWD}:/repo ghcr.io/opencyphal/toolshed:ts22.4.x
5069
```
70+
71+
### Push
72+
73+
As with load, you need to re-build with a `--push` argument but you'll be using the cache so the build should be a no-op:
74+
75+
```bash
76+
docker buildx build --platform linux/amd64,linux/arm64 --push -t ghcr.io/opencyphal/toolshed:ts22.4.x .
77+
```
78+
79+
## More on Multi-Platform Builders
80+
81+
The two commands above make some assumptions about defaults and capabilities that we haven't verified on all build hosts. First, the `buildx create` command is assumed to target the correct Docker context. You can see your contexts by doing:
82+
83+
```
84+
docker context ls
85+
```
86+
87+
... then target a specific context by adding it as an additional argument to the builder create command:
88+
89+
```
90+
docker buildx create --use --name cyphalbuild desktop-linux
91+
```
92+
93+
We also assume you are using a build that has our two supported host platforms `linux/amd64` and `linux/arm64`. You can verify this after creating the builder using the inspect command. This should also verify that your builder is now in effect:
94+
95+
```
96+
docker buildx inspect --bootstrap
97+
```

‎toolshed/gcc-arm-11.2-2022.02-aarch64-arm-none-eabi.sha512.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
be19622e61d320427143c8492d8955447b75de080f3228766942e3529874adcf1f174a84e8f1f3ec92d14f72e9a93ba3fc43a347000fecb1c4bf1c3c7cf0667f gcc-arm-11.2-2022.02-aarch64-arm-none-eabi.tar.xz

‎toolshed/gcc-arm-11.2-2022.02-aarch64-arm-none-linux-gnueabihf.sha512.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
f2acde25fbd5a0e5fcf2eb796942a7fa988dfe2a6190a0505f3b43b57ea3a4eb3b23e51aa35894d551a07a85561374040456826e407a02443daab132937f3be4 gcc-arm-11.2-2022.02-aarch64-arm-none-linux-gnueabihf.tar.xz

0 commit comments

Comments
 (0)