Skip to content

CI: Separate builds with native runners #277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
victorlin merged 3 commits into from
Jan 7, 2026
Merged

CI: Separate builds with native runners #277

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c5fd5a1
Conditionally prepend platform info in output
victorlin Jan 6, 2026
1ddb6e5
Define variables separately
victorlin Jan 6, 2026
52477eb
Separate builds with native runners
victorlin Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 74 additions & 24 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,22 @@ jobs:
cache-date: ${{ env.CACHE_DATE }}
tag: ${{ env.TAG }}

# Build multi-platform builder and final images with caching from Docker Hub
# and GitHub Container Registry; push to GitHub Container Registry.
# Build platform-specific builder and final images with caching from Docker
# Hub and GitHub Container Registry; push to GitHub Container Registry.
build:
name: build (${{ matrix.platform }})
needs: vars
runs-on: ubuntu-latest
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
arch: amd64
runner: ubuntu-latest

- platform: linux/arm64
arch: arm64
runner: ubuntu-24.04-arm
steps:

- uses: actions/checkout@v6
Expand All @@ -50,8 +61,6 @@ jobs:
with:
python-version: '>=3.8'

- uses: docker/setup-qemu-action@v3

# GITHUB_TOKEN is unreliable¹ so use a token from nextstrain-bot.
# ¹ https://github.com/docker/build-push-action/issues/463#issuecomment-939394233
- uses: docker/login-action@v3
Expand All @@ -60,7 +69,12 @@ jobs:
username: nextstrain-bot
password: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_MANAGE_PACKAGES }}

- run: ./devel/build -p linux/amd64,linux/arm64 -r ghcr.io -t "$TAG" -l logs/
- run: |
./devel/build \
-p ${{ matrix.platform }} \
-r ghcr.io \
-t "$TAG-${{ matrix.arch }}" \
-l logs-${{ matrix.arch }}/
env:
TAG: ${{ needs.vars.outputs.tag }}
CACHE_DATE: ${{ needs.vars.outputs.cache-date }}
Expand All @@ -69,15 +83,15 @@ jobs:
name: Upload build logs as artifacts
uses: actions/upload-artifact@v6
with:
name: build-logs
path: logs/
name: build-logs-${{ matrix.arch }}
path: logs-${{ matrix.arch }}/

- if: always()
name: Summarize build logs for the GitHub Actions run summary page
run: |
for log in logs/*; do
for log in logs-${{ matrix.arch }}/*; do
{
echo "## $(basename "$log")"
echo "## [${{ matrix.platform }}] $(basename "$log")"
echo ''
echo '```'
./devel/summarize-buildkit-output "$log" 2>&1
Expand All @@ -86,16 +100,55 @@ jobs:
} >> "$GITHUB_STEP_SUMMARY"
done

# Merge platform-specific images into multi-platform images
merge-builds:
needs: [vars, build]
runs-on: ubuntu-latest
env:
TAG: ${{ needs.vars.outputs.tag }}
steps:
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: nextstrain-bot
password: ${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_MANAGE_PACKAGES }}

- name: Create base-builder-build-platform image
run: |
docker buildx imagetools create \
-t "ghcr.io/nextstrain/base-builder-build-platform:$TAG" \
"ghcr.io/nextstrain/base-builder-build-platform:$TAG-amd64" \
"ghcr.io/nextstrain/base-builder-build-platform:$TAG-arm64"

- name: Create base-builder-target-platform image
run: |
docker buildx imagetools create \
-t "ghcr.io/nextstrain/base-builder-target-platform:$TAG" \
"ghcr.io/nextstrain/base-builder-target-platform:$TAG-amd64" \
"ghcr.io/nextstrain/base-builder-target-platform:$TAG-arm64"

- name: Create base image
run: |
docker buildx imagetools create \
-t "ghcr.io/nextstrain/base:$TAG" \
"ghcr.io/nextstrain/base:$TAG-amd64" \
"ghcr.io/nextstrain/base:$TAG-arm64"
Comment on lines +116 to +135
Copy link
Contributor

@joverlee521 joverlee521 Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking

The general pattern in this repo seems to run docker buildx commands in /devel/ scripts, should these steps be wrapped in a separate script?

Copy link
Member Author

@victorlin victorlin Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I'll keep it as is for a couple reasons:

  1. The README instructions for building locally using the existing devel scripts still work fine. The split build + merge-builds pattern is an optimization specific to the GitHub Actions workflow.
  2. Since these commands are simple, keeping them inline and split across 3 steps feels more readable, both in the code and on the GitHub run page. If it ever gets more complex, we can move to a devel wrapper script.


# Run tests with the final image from GitHub Container Registry.
test:
name: test (${{ matrix.platform }})
needs: [vars, build]
runs-on: ubuntu-latest
needs: [vars, merge-builds]
runs-on: ${{ matrix.runner }}
strategy:
matrix:
platform:
- linux/amd64
- linux/arm64
include:
- platform: linux/amd64
runner: ubuntu-latest
- platform: linux/arm64
# TODO: use ubuntu-24.04-arm and remove emulation when Nextstrain
# CLI can be installed natively on aarch64.¹
# ¹ <https://github.com/nextstrain/cli/pull/490>
runner: ubuntu-latest
permissions:
contents: read
id-token: write
Expand All @@ -116,9 +169,7 @@ jobs:
with:
python-version: ~3

# The ubuntu-latest runner is linux/amd64 so anything else will
# run with emulation, which is still better than nothing.
- if: matrix.platform != 'linux/amd64'
- if: matrix.platform == 'linux/arm64'
uses: docker/setup-qemu-action@v3

- uses: actions/checkout@v6
Expand All @@ -141,7 +192,7 @@ jobs:

validate-platforms:
name: Validate platforms
needs: [vars, build]
needs: [vars, merge-builds]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand All @@ -161,7 +212,7 @@ jobs:
# Docker Hub, where they will persist. Do this regardless of test results.
push-branch:
if: startsWith(needs.vars.outputs.tag, 'branch-') && github.event_name != 'pull_request'
needs: [vars, build]
needs: [vars, merge-builds]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand All @@ -185,7 +236,7 @@ jobs:
# Docker Hub, where they will persist. Only do this if tests pass.
push-build:
if: startsWith(needs.vars.outputs.tag, 'build-')
needs: [vars, build, test, validate-platforms]
needs: [vars, merge-builds, test, validate-platforms]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand Down Expand Up @@ -283,7 +334,7 @@ jobs:
# Delete the builder and final images from GitHub Container Registry.
cleanup-registry:
if: always()
needs: [vars, build, test, validate-platforms, push-branch, push-build]
needs: [vars, merge-builds, test, validate-platforms, push-branch, push-build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
Expand All @@ -294,5 +345,4 @@ jobs:
script: |
const script = require('./devel/delete-from-ghcr.js');
const tag = "${{ needs.vars.outputs.tag }}";
const token = "${{ secrets.GH_TOKEN_NEXTSTRAIN_BOT_MANAGE_PACKAGES }}";
script({fetch, octokit: github, tag, token});
script({octokit: github, tag});
99 changes: 35 additions & 64 deletions devel/delete-from-ghcr.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// ¹ https://github.com/octokit/core.js#authentication
// ² https://github.com/settings/tokens/new?scopes=delete:packages

module.exports = async ({fetch, octokit, tag, token}) => {
module.exports = async ({octokit, tag}) => {
org = 'nextstrain';
packages = [
'base',
Expand Down Expand Up @@ -35,86 +35,57 @@ module.exports = async ({fetch, octokit, tag, token}) => {
continue;
}

// The manifest list + one manifest per platform are pushed from the build.
// Only the manifest list is tagged for direct removal via GitHub's REST API.
async function deleteImage(tag) {
const versionsWithTag = packageVersions.filter(version => version.metadata.container.tags.includes(tag));

// The GitHub REST API does not provide a way to retrieve manifest digests
// from the manifest list, so use GHCR's (undocumented) Docker Registry API
// to do that.
// This works when a GitHub token with the right permissions is passed in
// base64 form as a Bearer token¹.
// ¹ https://github.com/orgs/community/discussions/26279#discussioncomment-3251172
const res = await fetch(`https://ghcr.io/v2/${org}/${packageName}/manifests/${tag}`,
{
headers: {
Accept: "application/vnd.docker.distribution.manifest.list.v2+json",
Authorization: `Bearer ${btoa(token)}`
}
});
const resData = await res.json();
const manifestDigests = resData.manifests.map(manifest => manifest.digest);
if (versionsWithTag.length == 0) {
console.error(`${org}/${packageName}:${tag} was not found.`);
errorEncountered = true;
return;
}

const versions = packageVersions.filter(version => manifestDigests.includes(version.name));
for (const version of versions) {
console.log(`Deleting the package version for ${org}/${packageName}:${version.name} ...`);
// Each tag should only correspond to one package version.
// Pushing an existing tag will untag the existing version and add the tag
// to the newly pushed version.
const versionId = versionsWithTag[0].id;
console.log(`Version for ${org}/${packageName}:${tag} is ${versionId}.`);

console.log(`Deleting the package version for ${org}/${packageName}:${tag} ...`);
try {
await octokit.request('DELETE /orgs/{org}/packages/{package_type}/{package_name}/versions/{package_version_id}', {
org: org,
package_type: 'container',
package_name: packageName,
package_version_id: version.id,
package_version_id: versionId,
});
console.log("Done.");
} catch (deleteVersionError) {
console.log(deleteVersionError);
errorEncountered = true;
continue;
}
}

// Delete the manifest list after deleting individual manifests.
if (deleteVersionError.response.data.message == "You cannot delete the last tagged version of a package. You must delete the package instead.") {

const versionsWithTag = packageVersions.filter(version => version.metadata.container.tags.includes(tag));
// The right thing to do would be to delete the package
// ${org}/${packageName}. However, this is a potential cause for
// transient 403 errors¹ on GitHub Actions, so we'll keep one tagged
// version around as a workaround until the underlying issue is fixed.
// ¹ https://github.com/nextstrain/docker-base/issues/131
console.log(`Not deleting ${org}/${packageName}:${tag} since that requires deleting the package.`);

if (versionsWithTag.length == 0) {
console.error(`${org}/${packageName}:${tag} was not found.`);
errorEncountered = true;
continue;
} else {
console.error(`Could not delete ${org}/${packageName}:${tag}.`);
errorEncountered = true;
}
}
}

// Each tag should only correspond to one package version.
// Pushing an existing tag will untag the existing version and add the tag
// to the newly pushed version.
versionId = versionsWithTag[0].id;
console.log(`Version for ${org}/${packageName}:${tag} is ${versionId}.`);

console.log(`Deleting the package version for ${org}/${packageName}:${tag} ...`);
try {
await octokit.request('DELETE /orgs/{org}/packages/{package_type}/{package_name}/versions/{package_version_id}', {
org: org,
package_type: 'container',
package_name: packageName,
package_version_id: versionId,
});
console.log("Done.");
} catch (deleteVersionError) {
console.log(deleteVersionError);

if (deleteVersionError.response.data.message == "You cannot delete the last tagged version of a package. You must delete the package instead.") {

// The right thing to do would be to delete the package
// ${org}/${packageName}. However, this is a potential cause for
// transient 403 errors¹ on GitHub Actions, so we'll keep one tagged
// version around as a workaround until the underlying issue is fixed.
// ¹ https://github.com/nextstrain/docker-base/issues/131
console.log(`Not deleting ${org}/${packageName}:${tag} since that requires deleting the package.`);

} else {
console.error(`Could not delete ${org}/${packageName}:${tag}.`);
errorEncountered = true;
continue;
}
// Delete platform-specific images.
const platforms = ["amd64", "arm64"];
for (const platform of platforms) {
await deleteImage(`${tag}-${platform}`);
}

// Delete the multi-platform image.
await deleteImage(tag);
}

if (errorEncountered) {
Expand Down