Skip to content

Commit 45fd741

Browse files
general-kroll-4-lifegeneral-kroll-4-life
committed
docker-build-parallel
Summary: 1. Due to shortfall in `docker` build tooling, it is not possible to first build and store a multi-architecture image and then test it and then push it. 2. Because point (1) is exactly what we require, and the prior push step has to halt and wait for `arm64`, time gets wasted. 3. This change seeks to mitigate same by storing a `tar` archive for each architecture, then testing on native, then retrieving the tar, and finally pushing multi-architecture. **Note**: unfortunately, at this time, `linux/arm64` GHA runners are not first class citizens. Once they are, there is significant potential improvement in build time compared to current QEMU pattern.
1 parent a891a31 commit 45fd741

File tree

2 files changed

+203
-41
lines changed

2 files changed

+203
-41
lines changed

‎.github/workflows/build.yml‎

Lines changed: 195 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1069,11 +1069,27 @@ jobs:
10691069
with:
10701070
name: stackql_darwin_arm64
10711071
path: build/stackql
1072-
1072+
1073+
## Docker Build and Push Jobs
1074+
## based loosely on patterns described in:
1075+
## - https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners
1076+
## - https://docs.docker.com/build/ci/github-actions/share-image-jobs/
1077+
##
1078+
## NOTE: The QEMU build for linux/arm64 is very slow. On the order of 30 minutes. This is currently unavoidable.
1079+
##
1080+
## TODO: Migrate linux/arm64 docker build to native once GHA supports this platform as a first class citizen.
1081+
##
10731082
dockerbuild:
10741083
name: Docker Build
10751084
runs-on: ubuntu-latest-m
10761085
timeout-minutes: ${{ vars.DEFAULT_JOB_TIMEOUT_MIN == '' && 120 || vars.DEFAULT_JOB_TIMEOUT_MIN }}
1086+
strategy:
1087+
fail-fast: false
1088+
matrix:
1089+
platform:
1090+
- linux/amd64
1091+
- linux/arm64
1092+
10771093
steps:
10781094

10791095
- name: Check out code into the Go module directory
@@ -1100,17 +1116,40 @@ jobs:
11001116
echo "SOURCE_TAG=${GITHUB_REF#refs/tags/}"
11011117
} >> "${GITHUB_STATE}"
11021118
1103-
- name: Install psql
1104-
run: |
1105-
sudo apt-get update
1106-
sudo apt-get install --yes --no-install-recommends \
1107-
postgresql-client \
1108-
ca-certificates \
1109-
openssl
1110-
1111-
- name: Install Python dependencies
1119+
- name: Image env sanitize
11121120
run: |
1113-
pip3 install -r cicd/requirements.txt
1121+
BUILD_IMAGE_REQUIRED="true"
1122+
PUSH_IMAGE_REQUIRED="false"
1123+
if [ "$( grep '^build-elide.*' <<< '${{ github.ref_name }}' )" != "" ]; then
1124+
BUILD_IMAGE_REQUIRED="false"
1125+
fi
1126+
# shellcheck disable=SC2235
1127+
if ( \
1128+
[ "${{ github.repository }}" = "stackql/stackql" ] \
1129+
|| [ "${{ github.repository }}" != "stackql/stackql-devel" ] \
1130+
) \
1131+
&& [ "${{ vars.CI_SKIP_DOCKER_PUSH }}" != "true" ] \
1132+
&& [ "$( grep '^build-elide.*' <<< '${{ github.ref_name }}' )" = "" ] \
1133+
&& ( \
1134+
[ "${{ github.ref_type }}" = "branch" ] \
1135+
&& [ "${{ github.ref_name }}" = "main" ] \
1136+
&& [ "${{ github.event_name }}" = "push" ] \
1137+
) \
1138+
|| ( \
1139+
[ "${{ github.ref_type }}" = "tag" ] \
1140+
&& [ "$( grep '^build-release.*' <<< '${{ github.ref_name }}' )" != "" ] \
1141+
); \
1142+
then
1143+
PUSH_IMAGE_REQUIRED="true"
1144+
fi
1145+
if [ "${{ matrix.platform }}" == "linux/arm64" ] && [ "${PUSH_IMAGE_REQUIRED}" = "false" ]; then
1146+
BUILD_IMAGE_REQUIRED="false"
1147+
fi
1148+
{
1149+
echo "IMAGE_PLATFORM_SAN=$( sed 's/\//_/g' <<< '${{ matrix.platform }}' )";
1150+
echo "PUSH_IMAGE_REQUIRED=${PUSH_IMAGE_REQUIRED}";
1151+
echo "BUILD_IMAGE_REQUIRED=${BUILD_IMAGE_REQUIRED}";
1152+
} | tee -a "${GITHUB_ENV}"
11141153
11151154
- name: Extract Build Info and Persist
11161155
env:
@@ -1146,20 +1185,38 @@ jobs:
11461185
echo "GID=${GID}"
11471186
} >> "${GITHUB_ENV}"
11481187
1188+
- name: Install psql
1189+
if: env.BUILD_IMAGE_REQUIRED == 'true'
1190+
run: |
1191+
sudo apt-get update
1192+
sudo apt-get install --yes --no-install-recommends \
1193+
postgresql-client \
1194+
ca-certificates \
1195+
openssl
1196+
1197+
# for some reason skipping this with env.BUILD_IMAGE_REQUIRED == 'true' breaks python cleanup where it can't find pip cache
1198+
- name: Install Python dependencies
1199+
run: |
1200+
pip3 install -r cicd/requirements.txt
1201+
11491202
- name: Generate rewritten registry for simulations
1203+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11501204
run: |
11511205
python3 test/python/registry-rewrite.py --replacement-host=host.docker.internal
11521206
11531207
- name: Pull Docker base images for cache purposes
1208+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11541209
run: |
11551210
docker pull golang:1.18.4-bullseye
11561211
docker pull ubuntu:22.04
11571212
11581213
- name: Pull Docker image for cache purposes
1214+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11591215
run: |
11601216
docker pull stackql/stackql:latest || echo 'could not pull image for cache purposes'
11611217
11621218
- name: Create certificates for robot tests
1219+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11631220
run: |
11641221
openssl req -x509 -keyout test/server/mtls/credentials/pg_server_key.pem -out test/server/mtls/credentials/pg_server_cert.pem -config test/server/mtls/openssl.cnf -days 365
11651222
openssl req -x509 -keyout test/server/mtls/credentials/pg_client_key.pem -out test/server/mtls/credentials/pg_client_cert.pem -config test/server/mtls/openssl.cnf -days 365
@@ -1168,26 +1225,43 @@ jobs:
11681225
openssl req -x509 -keyout cicd/vol/srv/credentials/pg_client_key.pem -out cicd/vol/srv/credentials/pg_client_cert.pem -config test/server/mtls/openssl.cnf -days 365
11691226
openssl req -x509 -keyout cicd/vol/srv/credentials/pg_rubbish_key.pem -out cicd/vol/srv/credentials/pg_rubbish_cert.pem -config test/server/mtls/openssl.cnf -days 365
11701227
1171-
- name: Build image
1228+
- name: Build image precursors
1229+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11721230
run: |
11731231
docker compose -f docker-compose-credentials.yml build credentialsgen
11741232
docker compose build mockserver
11751233
11761234
- name: Build Stackql image with buildx
1177-
uses: docker/build-push-action@v5
1235+
uses: docker/build-push-action@v6
1236+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11781237
with:
11791238
context: .
11801239
build-args: |
11811240
BUILDMAJORVERSION=${{env.BUILDMAJORVERSION}}
11821241
BUILDMINORVERSION=${{env.BUILDMINORVERSION}}
11831242
BUILDPATCHVERSION=${{env.BUILDPATCHVERSION}}
11841243
push: false
1244+
platforms: ${{ matrix.platform }}
11851245
target: app
11861246
no-cache: ${{ vars.CI_DOCKER_BUILD_NO_CACHE == 'true' && true || false }}
1187-
load: true
1188-
tags: ${{ env.STACKQL_IMAGE_NAME }}:${{github.sha}},${{ env.STACKQL_IMAGE_NAME }}:v${{env.BUILDMAJORVERSION}}.${{env.BUILDMINORVERSION}}.${{env.BUILDPATCHVERSION}},${{ env.STACKQL_IMAGE_NAME }}:latest
1247+
tags: ${{ env.STACKQL_IMAGE_NAME }}:${{ github.sha }},${{ env.STACKQL_IMAGE_NAME }}:v${{env.BUILDMAJORVERSION}}.${{env.BUILDMINORVERSION}}.${{env.BUILDPATCHVERSION}},${{ env.STACKQL_IMAGE_NAME }}
1248+
outputs: type=docker,dest=${{ runner.temp }}/myimage-${{ env.IMAGE_PLATFORM_SAN }}.tar
1249+
1250+
- name: Upload artifact
1251+
uses: actions/upload-artifact@v4
1252+
if: env.BUILD_IMAGE_REQUIRED == 'true'
1253+
with:
1254+
name: myimage-${{ env.IMAGE_PLATFORM_SAN }}
1255+
path: ${{ runner.temp }}/myimage-${{ env.IMAGE_PLATFORM_SAN }}.tar
1256+
1257+
- name: Load image
1258+
if: env.BUILD_IMAGE_REQUIRED == 'true'
1259+
run: |
1260+
docker load --input ${{ runner.temp }}/myimage-${{ env.IMAGE_PLATFORM_SAN }}.tar
1261+
docker image ls -a
11891262
11901263
- name: Debug info
1264+
if: env.BUILD_IMAGE_REQUIRED == 'true'
11911265
run: |
11921266
echo "psql version info: $(psql --version)"
11931267
echo ""
@@ -1223,13 +1297,13 @@ jobs:
12231297
echo ""
12241298
12251299
- name: Run robot mocked functional tests
1226-
if: success() && env.CI_IS_EXPRESS != 'true'
1300+
if: success() && env.CI_IS_EXPRESS != 'true' && matrix.platform == 'linux/amd64' && env.BUILD_IMAGE_REQUIRED == 'true'
12271301
timeout-minutes: ${{ vars.DEFAULT_STEP_TIMEOUT_MIN == '' && 20 || vars.DEFAULT_STEP_TIMEOUT_MIN }}
12281302
run: |
12291303
python cicd/python/build.py --robot-test --config='{ "variables": { "EXECUTION_PLATFORM": "docker" } }'
12301304
12311305
- name: Run POSTGRES BACKEND robot mocked functional tests
1232-
if: success() && env.CI_IS_EXPRESS != 'true'
1306+
if: success() && env.CI_IS_EXPRESS != 'true' && matrix.platform == 'linux/amd64' && env.BUILD_IMAGE_REQUIRED == 'true'
12331307
timeout-minutes: ${{ vars.DEFAULT_STEP_TIMEOUT_MIN == '' && 20 || vars.DEFAULT_STEP_TIMEOUT_MIN }}
12341308
run: |
12351309
echo "## Stray flask apps to be killed before robot tests ##"
@@ -1249,12 +1323,12 @@ jobs:
12491323
python cicd/python/build.py --robot-test --config='{ "variables": { "EXECUTION_PLATFORM": "docker", "SHOULD_RUN_DOCKER_EXTERNAL_TESTS": true, "SQL_BACKEND": "postgres_tcp" } }'
12501324
12511325
- name: Output from mocked functional tests
1252-
if: always() && env.CI_IS_EXPRESS != 'true'
1326+
if: always() && env.CI_IS_EXPRESS != 'true' && matrix.platform == 'linux/amd64' && env.BUILD_IMAGE_REQUIRED == 'true'
12531327
run: |
12541328
cat ./test/robot/reports/output.xml
12551329
12561330
- name: Run robot integration tests
1257-
if: env.AZURE_CLIENT_SECRET != '' && startsWith(env.STATE_SOURCE_TAG, 'build-release')
1331+
if: env.AZURE_CLIENT_SECRET != '' && startsWith(env.STATE_SOURCE_TAG, 'build-release') && matrix.platform == 'linux/amd64' && env.BUILD_IMAGE_REQUIRED == 'true'
12581332
env:
12591333
AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
12601334
AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
@@ -1266,30 +1340,113 @@ jobs:
12661340
echo "## End ##"
12671341
python cicd/python/build.py --robot-test-integration --config='{ "variables": { "EXECUTION_PLATFORM": "docker" } }'
12681342
1343+
# - name: Hack to avoid docker buildx failures
1344+
# run: |
1345+
# sudo rm -rf cicd/vol/postgres/persist
1346+
1347+
dockerpush:
1348+
runs-on: ubuntu-latest
1349+
needs: dockerbuild
1350+
strategy:
1351+
fail-fast: false
1352+
matrix:
1353+
platform:
1354+
- linux/amd64
1355+
- linux/arm64
1356+
steps:
1357+
1358+
- name: Check out code into the Go module directory
1359+
uses: actions/[email protected]
1360+
1361+
- name: Image env sanitize
1362+
run: |
1363+
BUILD_IMAGE_REQUIRED="true"
1364+
PUSH_IMAGE_REQUIRED="false"
1365+
if [ "$( grep '^build-elide.*' <<< '${{ github.ref_name }}' )" != "" ]; then
1366+
BUILD_IMAGE_REQUIRED="false"
1367+
fi
1368+
# shellcheck disable=SC2235
1369+
if ( \
1370+
[ "${{ github.repository }}" = "stackql/stackql" ] \
1371+
|| [ "${{ github.repository }}" != "stackql/stackql-devel" ] \
1372+
) \
1373+
&& [ "${{ vars.CI_SKIP_DOCKER_PUSH }}" != "true" ] \
1374+
&& [ "$( grep '^build-elide.*' <<< '${{ github.ref_name }}' )" = "" ] \
1375+
&& ( \
1376+
[ "${{ github.ref_type }}" = "branch" ] \
1377+
&& [ "${{ github.ref_name }}" = "main" ] \
1378+
&& [ "${{ github.event_name }}" = "push" ] \
1379+
) \
1380+
|| ( \
1381+
[ "${{ github.ref_type }}" = "tag" ] \
1382+
&& [ "$( grep '^build-release.*' <<< '${{ github.ref_name }}' )" != "" ] \
1383+
); \
1384+
then
1385+
PUSH_IMAGE_REQUIRED="true"
1386+
fi
1387+
if [ "${{ matrix.platform }}" == "linux/arm64" ] && [ "${PUSH_IMAGE_REQUIRED}" = "false" ]; then
1388+
BUILD_IMAGE_REQUIRED="false"
1389+
fi
1390+
{
1391+
echo "IMAGE_PLATFORM_SAN=$( sed 's/\//_/g' <<< '${{ matrix.platform }}' )";
1392+
echo "PUSH_IMAGE_REQUIRED=${PUSH_IMAGE_REQUIRED}";
1393+
echo "BUILD_IMAGE_REQUIRED=${BUILD_IMAGE_REQUIRED}";
1394+
} | tee -a "${GITHUB_ENV}"
1395+
1396+
- name: Download artifact
1397+
uses: actions/download-artifact@v4
1398+
if: env.PUSH_IMAGE_REQUIRED == 'true'
1399+
with:
1400+
name: myimage-${{ env.IMAGE_PLATFORM_SAN }}
1401+
path: ${{ runner.temp }}
1402+
1403+
- name: Extract Build Info and Persist
1404+
env:
1405+
BUILDCOMMITSHA: ${{github.sha}}
1406+
BUILDBRANCH: ${{github.ref}}
1407+
BUILDPLATFORM: ${{runner.os}}
1408+
BUILDPATCHVERSION: ${{github.run_number}}
1409+
run: |
1410+
source cicd/version.txt
1411+
BUILDMAJORVERSION=${MajorVersion}
1412+
BUILDMINORVERSION=${MinorVersion}
1413+
if [[ ! "$BUILDBRANCH" == "*develop" ]]; then
1414+
# shellcheck disable=2269
1415+
BUILDPATCHVERSION="${BUILDPATCHVERSION}"
1416+
fi
1417+
BUILDSHORTCOMMITSHA="$(echo "${BUILDCOMMITSHA}" | cut -c 1-7)"
1418+
BUILDDATE="$(date)"
1419+
export BUILDDATE
1420+
echo "BUILDMAJORVERSION: ${BUILDMAJORVERSION}"
1421+
echo "BUILDMINORVERSION: ${BUILDMINORVERSION}"
1422+
echo "BUILDPATCHVERSION: ${BUILDPATCHVERSION}"
1423+
echo "BUILDBRANCH: ${BUILDBRANCH}"
1424+
echo "BUILDCOMMITSHA: ${BUILDCOMMITSHA}"
1425+
echo "BUILDSHORTCOMMITSHA: ${BUILDSHORTCOMMITSHA}"
1426+
echo "BUILDDATE: ${BUILDDATE}"
1427+
echo "BUILDPLATFORM: ${BUILDPLATFORM}"
1428+
1429+
{
1430+
echo "BUILDMAJORVERSION=$BUILDMAJORVERSION"
1431+
echo "BUILDMINORVERSION=$BUILDMINORVERSION"
1432+
echo "BUILDPATCHVERSION=$BUILDPATCHVERSION"
1433+
echo "UID=${UID}"
1434+
echo "GID=${GID}"
1435+
} >> "${GITHUB_ENV}"
1436+
12691437
- name: Login to Docker Hub
1270-
if: ${{ ( success() && github.ref_type == 'branch' && github.ref_name == 'main' && github.repository == 'stackql/stackql' && github.event_name == 'push' ) || ( success() && github.ref_type == 'tag' && startsWith(github.ref_name, 'build-release') ) }}
1438+
if: env.PUSH_IMAGE_REQUIRED == 'true'
12711439
uses: docker/login-action@v2
12721440
with:
12731441
username: ${{ secrets.DOCKERHUB_USERNAME }}
12741442
password: ${{ secrets.DOCKERHUB_TOKEN }}
12751443

1276-
- name: Hack to avoid docker buildx failures
1444+
- name: Load image
1445+
if: env.PUSH_IMAGE_REQUIRED == 'true'
12771446
run: |
1278-
sudo rm -rf cicd/vol/postgres/persist
1279-
1280-
- name: Push stackql image to Docker Hub
1281-
if: ${{ (github.repository == 'stackql/stackql' || github.repository == 'stackql/stackql-devel') && vars.CI_SKIP_DOCKER_PUSH != 'true' && ( success() && github.ref_type == 'branch' && github.ref_name == 'main' && github.event_name == 'push' ) || ( success() && github.ref_type == 'tag' && startsWith(github.ref_name, 'build-release') ) }}
1282-
uses: docker/build-push-action@v5
1283-
with:
1284-
context: .
1285-
no-cache: ${{ vars.CI_DOCKER_BUILD_NO_CACHE == 'true' && true || false }}
1286-
platforms: linux/arm64,linux/amd64
1287-
build-args: |
1288-
BUILDMAJORVERSION=${{env.BUILDMAJORVERSION}}
1289-
BUILDMINORVERSION=${{env.BUILDMINORVERSION}}
1290-
BUILDPATCHVERSION=${{env.BUILDPATCHVERSION}}
1291-
RUN_INTEGRATION_TESTS=0
1292-
push: true
1293-
target: app
1294-
tags: ${{ env.STACKQL_IMAGE_NAME }}:${{github.sha}},${{ env.STACKQL_IMAGE_NAME }}:v${{env.BUILDMAJORVERSION}}.${{env.BUILDMINORVERSION}}.${{env.BUILDPATCHVERSION}},${{ env.STACKQL_IMAGE_NAME }}:latest
1447+
docker load --input ${{ runner.temp }}/myimage-${{ env.IMAGE_PLATFORM_SAN }}.tar
1448+
docker image ls -a
1449+
docker push ${{ env.STACKQL_IMAGE_NAME }}:${{github.sha}} && \
1450+
docker push ${{ env.STACKQL_IMAGE_NAME }}:v${{env.BUILDMAJORVERSION}}.${{env.BUILDMINORVERSION}}.${{env.BUILDPATCHVERSION}} && \
1451+
docker push ${{ env.STACKQL_IMAGE_NAME }}:latest
12951452

‎docs/CICD.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,15 @@
44

55
Summary:
66

7-
- At present, PR checks, build and test are all performed through [.github/workflows/go.yml](/.github/workflows/go.yml).
7+
- At present, PR checks, build and test are all performed through [.github/workflows/build.yml](/.github/workflows/build.yml).
88
- Releasing over various channels (website, homebrew, chocolatey...) is performed manually.
9-
- The strategic state is to split the functions: PR checks, build and test; into separate files, and migrate to use [goreleaser](https://goreleaser.com/).
10-
- Should take the hint from docker to [speed up multi-platform builds using multiple runners](https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners).
9+
- ~~The strategic state is to split the functions: PR checks, build and test; into separate files, and migrate to use [goreleaser](https://goreleaser.com/).~~
10+
- Docker Build and Push Jobs have scope for improvement.
11+
- These are currently based loosely on patterns described in:
12+
- https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners
13+
- https://docs.docker.com/build/ci/github-actions/share-image-jobs/
14+
- **NOTE**: The QEMU build for linux/arm64 is **very slow**. On the order of 30 minutes. This is currently unavoidable.
15+
- **TODO**: Migrate linux/arm64 docker build to native once GHA supports this platform as a first class citizen.
1116

1217

1318
## Secrets

0 commit comments

Comments
 (0)