Skip to content

Commit dbaf6ed

Browse files
thirdeyenickthirdeyenick
committed
improve CI workflow
This improves the CI workflow for the base images. Whenever we merge/push to `main`, we shellcheck and try to build the images (but we don't push them). Every night, `nightly` tagged images are created and pushed. Then, on a biweekly basis, we build the current image, based on `main` and compare the result to the latest `stable` image. We gather the changed ubuntu packages and release the just built image as the new `stable` one. A new tag is pushed and a github release is created which contains the changed packages and all commits added since the last tag which contain the conventinal commit preambles `fix`, `feat` or `breaking`.
1 parent d79c41e commit dbaf6ed

File tree

8 files changed

+459
-136
lines changed

8 files changed

+459
-136
lines changed

‎.github/workflows/build-and-publish-nightly.yml‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
name: Build and Publish Nightly
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
publish:
7+
description: "Whether to publish the built images as nightly"
8+
type: boolean
9+
default: true
10+
11+
permissions:
12+
packages: write
13+
14+
jobs:
15+
build:
16+
name: "Build deploio-heroku:${{ matrix.stack-version }} (${{ matrix.arch }}) image"
17+
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
arch: ["amd64", "arm64"]
22+
stack-version: ["24"]
23+
env:
24+
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
25+
steps:
26+
- name: Checkout
27+
uses: actions/checkout@v6
28+
- name: Build base image groups
29+
run: bin/build.sh "${{ matrix.stack-version }}" "${{ matrix.arch }}"
30+
- name: Export base images from the Docker daemon
31+
if: ${{ inputs.publish }}
32+
run: |
33+
docker save $(docker images --format '{{.Repository}}:{{.Tag}}' | grep "${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}") | zstd -T0 --long=31 -o images.tar.zst
34+
- name: Save base image exports to the cache
35+
if: ${{ inputs.publish }}
36+
uses: actions/cache/save@v5
37+
with:
38+
key: ${{ github.run_id }}-${{ matrix.stack-version }}-${{ matrix.arch }}
39+
path: images.tar.zst
40+
41+
publish-nightly-images:
42+
if: ${{ inputs.publish }}
43+
name: "Publish nightly deploio-heroku:${{ matrix.stack-version }} (${{ matrix.arch }}) image"
44+
needs:
45+
- build
46+
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
47+
strategy:
48+
fail-fast: false
49+
matrix:
50+
arch: ["amd64", "arm64"]
51+
stack-version: ["24"]
52+
env:
53+
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
54+
steps:
55+
- name: Restore base images from the cache
56+
uses: actions/cache/restore@v5
57+
with:
58+
fail-on-cache-miss: true
59+
key: ${{ github.run_id }}-${{ matrix.stack-version }}-${{ matrix.arch }}
60+
path: images.tar.zst
61+
env:
62+
SEGMENT_DOWNLOAD_TIMEOUT_MINS: 1
63+
- name: Load Docker images into the Docker daemon
64+
run: zstd -dc --long=31 images.tar.zst | docker load
65+
- name: Log in to GitHub Container Registry
66+
uses: docker/login-action@v3
67+
with:
68+
registry: ghcr.io
69+
username: ${{ github.actor }}
70+
password: ${{ secrets.GITHUB_TOKEN }}
71+
- name: Publish base images to registry
72+
run: |
73+
for variant in "" "-build"; do
74+
srcTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}"
75+
nightlyTag="${srcTag}_linux-${{ matrix.arch }}.nightly"
76+
docker tag "${srcTag}" "${nightlyTag}"
77+
docker push "${nightlyTag}"
78+
done
79+
80+
publish-nightly-multi-arch:
81+
if: ${{ inputs.publish }}
82+
name: "Publish nightly deploio-heroku:${{ matrix.stack-version }} multi-arch indices"
83+
needs:
84+
- publish-nightly-images
85+
runs-on: ubuntu-24.04
86+
env:
87+
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
88+
strategy:
89+
fail-fast: false
90+
matrix:
91+
stack-version: ["24"]
92+
steps:
93+
- name: Log in to GitHub Container Registry
94+
uses: docker/login-action@v3
95+
with:
96+
registry: ghcr.io
97+
username: ${{ github.actor }}
98+
password: ${{ secrets.GITHUB_TOKEN }}
99+
- name: Publish multi-arch image index
100+
run: |
101+
for variant in '' '-build'; do
102+
armTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-arm64.nightly"
103+
amdTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-amd64.nightly"
104+
nightlyIndex="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}.nightly"
105+
docker manifest create "${nightlyIndex}" "${amdTag}" "${armTag}"
106+
docker manifest push "${nightlyIndex}"
107+
done

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 133 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,6 @@ on:
55
# Avoid duplicate builds on PRs.
66
branches:
77
- main
8-
tags:
9-
- v*
10-
schedule:
11-
- cron: "0 0 * * 1-5"
128
pull_request:
139
workflow_dispatch:
1410

@@ -26,134 +22,8 @@ jobs:
2622
run: find . -type f \( -name "*.sh" -o -path "*/bin/*" \) ! -name '*.jq' | xargs -t shellcheck
2723

2824
build:
29-
name: "Build heroku-${{ matrix.stack-version }} (${{ matrix.arch }})"
3025
needs:
3126
- shellcheck
32-
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
33-
strategy:
34-
fail-fast: false
35-
matrix:
36-
arch: ["amd64", "arm64"]
37-
stack-version: ["24"]
38-
env:
39-
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
40-
steps:
41-
- name: Checkout
42-
uses: actions/checkout@v6
43-
- name: Build base image groups
44-
run: bin/build.sh "${{ matrix.stack-version }}" "${{ matrix.arch }}"
45-
- name: Check that the generated files are in sync
46-
run: |-
47-
status="$(git status --porcelain)"
48-
if [[ -n "$status" ]]; then
49-
echo "Generated files differ from checked-in versions! Run bin/build.sh to regenerate them locally."
50-
echo -e "\nChanged files:\n${status}\n"
51-
git diff
52-
exit 1
53-
fi
54-
- name: Export base images from the Docker daemon
55-
if: github.ref_name == 'main' || github.ref_type == 'tag'
56-
run: |
57-
docker save $(docker images --format '{{.Repository}}:{{.Tag}}' | grep "${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}") | zstd -T0 --long=31 -o images.tar.zst
58-
- name: Save OCI base image exports to the cache
59-
if: github.ref_name == 'main' || github.ref_type == 'tag'
60-
uses: actions/cache/save@v5
61-
with:
62-
key: ${{ github.run_id}}-${{ matrix.stack-version }}-${{ matrix.arch }}
63-
path: images.tar.zst
64-
65-
publish-images:
66-
if: github.ref_name == 'main' || github.ref_type == 'tag'
67-
name: "Publish heroku-${{ matrix.stack-version }} (${{ matrix.arch }})"
68-
needs:
69-
- build
70-
runs-on: ${{ matrix.arch == 'arm64' && 'ubuntu-24.04-arm' || 'ubuntu-24.04' }}
71-
env:
72-
TAG_SUFFIX: ".${{ github.ref_type == 'tag' && github.ref_name || 'nightly' }}"
73-
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
74-
strategy:
75-
fail-fast: false
76-
matrix:
77-
arch: ["amd64", "arm64"]
78-
stack-version: ["24"]
79-
steps:
80-
- name: Restore base images from the cache
81-
uses: actions/cache/restore@v5
82-
with:
83-
fail-on-cache-miss: true
84-
key: ${{ github.run_id}}-${{ matrix.stack-version }}-${{ matrix.arch }}
85-
path: images.tar.zst
86-
env:
87-
SEGMENT_DOWNLOAD_TIMEOUT_MINS: 1
88-
- name: Load Docker images into the Docker daemon
89-
run: zstd -dc --long=31 images.tar.zst | docker load
90-
- name: Log in to GitHub Container Registry
91-
run: echo '${{ secrets.GITHUB_TOKEN }}' | docker login ghcr.io -u '${{ github.actor }}' --password-stdin
92-
- name: Publish base images to registry
93-
run: |
94-
for variant in "" "-build"; do
95-
srcTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}"
96-
destTag="${srcTag}_linux-${{ matrix.arch }}${TAG_SUFFIX}"
97-
docker tag "${srcTag}" "${destTag}"
98-
docker push "${destTag}"
99-
done
100-
101-
publish-indices:
102-
if: github.ref_name == 'main' || github.ref_type == 'tag'
103-
name: "Publish heroku-${{ matrix.stack-version }} indices"
104-
needs:
105-
- publish-images
106-
runs-on: ubuntu-24.04
107-
env:
108-
TAG_SUFFIX: ".${{ github.ref_type == 'tag' && github.ref_name || 'nightly' }}"
109-
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
110-
strategy:
111-
fail-fast: false
112-
matrix:
113-
stack-version: ["24"]
114-
steps:
115-
- name: Log in to GitHub Container Registry
116-
run: echo '${{ secrets.GITHUB_TOKEN }}' | docker login ghcr.io -u '${{ github.actor }}' --password-stdin
117-
- name: Publish multi-arch image index
118-
run: |
119-
for variant in '' '-build'; do
120-
indexTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}${TAG_SUFFIX}"
121-
armTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-arm64${TAG_SUFFIX}"
122-
amdTag="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-amd64${TAG_SUFFIX}"
123-
docker manifest create "${indexTag}" "${amdTag}" "${armTag}"
124-
docker manifest push "${indexTag}"
125-
done
126-
127-
promote-tags:
128-
if: github.ref_type == 'tag'
129-
name: "Promote heroku-${{ matrix.stack-version }} tags"
130-
needs:
131-
- publish-images
132-
- publish-indices
133-
runs-on: ubuntu-24.04
134-
strategy:
135-
fail-fast: false
136-
matrix:
137-
stack-version: ["24"]
138-
env:
139-
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
140-
steps:
141-
- name: Checkout
142-
uses: actions/checkout@v6
143-
- name: Log in to GitHub Container Registry
144-
run: echo '${{ secrets.GITHUB_TOKEN }}' | docker login ghcr.io -u '${{ github.actor }}' --password-stdin
145-
- name: Install crane
146-
uses: buildpacks/github-actions/setup-tools@f3ec16c6d708761c6e87bbc8fe7f97375f80e7cd # v5.10.1
147-
- name: Promote images to stable tag
148-
run: |
149-
destTags=( )
150-
for variant in '' '-build'; do
151-
for arch in 'amd64' 'arm64'; do
152-
destTags+=("${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-${arch}")
153-
done
154-
destTags+=("${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}")
155-
done
156-
for destTag in "${destTags[@]}"; do
157-
srcTag="${destTag}.${{ github.ref_name }}"
158-
crane copy "${srcTag}" "${destTag}"
159-
done
27+
uses: ./.github/workflows/build-and-publish-nightly.yml
28+
with:
29+
publish: false

‎.github/workflows/nightly.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
name: Nightly
2+
3+
on:
4+
schedule:
5+
- cron: "0 3 * * *"
6+
workflow_dispatch:
7+
8+
permissions:
9+
packages: write
10+
11+
jobs:
12+
build-and-publish-nightly:
13+
uses: ./.github/workflows/build-and-publish-nightly.yml
14+
secrets: inherit

‎.github/workflows/release.yml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
name: Release
2+
3+
on:
4+
schedule:
5+
- cron: "0 7 1,15 * *"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: write
10+
packages: write
11+
12+
jobs:
13+
build-and-publish-nightly:
14+
uses: ./.github/workflows/build-and-publish-nightly.yml
15+
secrets: inherit
16+
17+
check-digest:
18+
name: "Check if release is needed"
19+
needs: build-and-publish-nightly
20+
runs-on: ubuntu-24.04
21+
outputs:
22+
should-release: ${{ steps.check.outputs.should-release }}
23+
next-tag: ${{ steps.check.outputs.next-tag }}
24+
from-tag: ${{ steps.check.outputs.from-tag }}
25+
env:
26+
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
27+
steps:
28+
- name: Checkout
29+
uses: actions/checkout@v6
30+
with:
31+
fetch-depth: 0
32+
- name: Install crane
33+
uses: buildpacks/github-actions/setup-tools@f3ec16c6d708761c6e87bbc8fe7f97375f80e7cd # v5.10.1
34+
- name: Log in to GitHub Container Registry
35+
uses: docker/login-action@v3
36+
with:
37+
registry: ghcr.io
38+
username: ${{ github.actor }}
39+
password: ${{ secrets.GITHUB_TOKEN }}
40+
- name: Check digests and determine if release is needed
41+
id: check
42+
run: bin/check-digest.sh
43+
44+
release:
45+
name: "Release deploio-heroku-${{ matrix.stack-version }}"
46+
needs: check-digest
47+
if: needs.check-digest.outputs.should-release == 'true'
48+
runs-on: ubuntu-24.04
49+
strategy:
50+
fail-fast: false
51+
matrix:
52+
stack-version: ["24"]
53+
env:
54+
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/deploio-heroku
55+
steps:
56+
- name: Checkout
57+
uses: actions/checkout@v6
58+
with:
59+
fetch-depth: 0
60+
- name: Install crane
61+
uses: buildpacks/github-actions/setup-tools@f3ec16c6d708761c6e87bbc8fe7f97375f80e7cd # v5.10.1
62+
- name: Log in to GitHub Container Registry
63+
uses: docker/login-action@v3
64+
with:
65+
registry: ghcr.io
66+
username: ${{ github.actor }}
67+
password: ${{ secrets.GITHUB_TOKEN }}
68+
- name: Generate release notes
69+
run: |
70+
bin/generate-release-notes.sh \
71+
"${{ matrix.stack-version }}" \
72+
"${DOCKER_IMAGE_NAME}" \
73+
"${{ needs.check-digest.outputs.from-tag }}" \
74+
> /tmp/release-notes.md
75+
- name: Promote nightly images to versioned and stable tags
76+
run: |
77+
nextTag="${{ needs.check-digest.outputs.next-tag }}"
78+
for variant in '' '-build'; do
79+
for arch in 'amd64' 'arm64'; do
80+
src="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-${arch}.nightly"
81+
versioned="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-${arch}.${nextTag}"
82+
stable="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}_linux-${arch}"
83+
crane copy "${src}" "${versioned}"
84+
crane copy "${src}" "${stable}"
85+
done
86+
src_index="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}.nightly"
87+
versioned_index="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}.${nextTag}"
88+
stable_index="${DOCKER_IMAGE_NAME}:${{ matrix.stack-version }}${variant}"
89+
crane copy "${src_index}" "${versioned_index}"
90+
crane copy "${src_index}" "${stable_index}"
91+
done
92+
- name: Create git tag and push
93+
run: |
94+
git config user.name "github-actions[bot]"
95+
git config user.email "github-actions[bot]@users.noreply.github.com"
96+
git tag "${{ needs.check-digest.outputs.next-tag }}"
97+
git push origin "${{ needs.check-digest.outputs.next-tag }}"
98+
- name: Create GitHub release
99+
env:
100+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
101+
run: gh release create "${{ needs.check-digest.outputs.next-tag }}" --title "${{ needs.check-digest.outputs.next-tag }}" --notes-file /tmp/release-notes.md

‎BUILD.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,12 @@ The `*-build` variants include all the packages from the non-build variant by de
3434

3535
We use GitHub Actions to build and release the base images:
3636

37-
* Any push to `main` will build the images and push the nightly GitHub Container Registry tag variants (such as `ghcr.io/ninech/deploio-heroku:24-build.nightly`).
38-
* Any new Git tag will build the image and push the latest GitHub Container Registry tag (such as `ghcr.io/ninech/deploio-heroku:24-build`),
39-
as well as a versioned tag (such as `ghcr.io/ninech/deploio-heroku:24-build.v123`).
37+
* Any push to `main` will build the images (proves that images can be build)
38+
* A nightly job pushes `nightly` tagged GitHub Container Registry variants (such as `ghcr.io/ninech/deploio-heroku:24-build.nightly`).
39+
* A special release pipeline will run every 2 weeks automatically and create a new `stable`
40+
tag of the `run` and `build` images. A new tag and github release will also be created automatically. This
41+
pipeline can also be started manually to quickly release a new `stable` version.
42+
43+
The message attached to the github release will contain all changed Ubuntu
44+
packages (and their versions) and messages made by the conventional commit
45+
preambles `fix`, `feat` and `breaking change`.

‎bin/build.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ write_package_list() {
7676
done
7777
}
7878

79+
7980
RUN_IMAGE_TAG="${REPO}:${STACK_VERSION}"
8081
RUN_DOCKERFILE_DIR="heroku-${STACK_VERSION}"
8182
[[ -d "${RUN_DOCKERFILE_DIR}" ]] || abort "fatal: directory ${RUN_DOCKERFILE_DIR} not found"

0 commit comments

Comments
 (0)