Merge pull request #159 from basecubedev/fix/appliance-home-identity-… #5

Workflow file for this run

.github/workflows/appliance-image.yml at a34aa05

	name: Appliance image build

	# Build the flashable Raspberry Pi appliance image on a GitHub-hosted runner.
	#
	# What this workflow produces is a build. Per board: an image an operator can
	# write to a card, the build authority that says what it is made of, the builder
	# environment it was recorded with, and the gate logs that say what was proven
	# about it. Once per run: the Appliance Manager package the images carry.
	#
	# A build every board finished is published as a GitHub prerelease, because
	# docs/user/appliance/install.md sends an operator to the Releases page for
	# exactly these file names, and an Actions artefact expires in thirty days
	# behind a sign-in wall, on a run page no operator will find. Prerelease is not
	# decoration: GitHub's /releases/latest alias skips one, so a weekly unsigned
	# build never becomes the "Latest" release this repository advertises.
	#
	# What it does not produce is a release, and that is deliberate.
	# packaging/appliance/vm/base-images.lock.json approves exactly one builder --
	# the pinned Debian genericcloud guest -- and tells it from every other machine
	# by a kernel pattern no hosted runner carries. An image built here is therefore
	# refused at signing time with builder_environment_untrusted, by design. Cutting
	# a signable release still means scripts/appliance-builder-vm.sh on a host booted
	# from that image. Nothing here is relaxed to make a gate pass: the builder
	# environment is recorded honestly, including the base-image digest a hosted
	# runner does not have, so a finalizer refuses it for the true reason.
	#
	# Flashing does not verify an image signature, so an unsigned build is enough for
	# the physical-hardware validation gate in docs/appliance/hardware-validation.md.

	"on":
	# Weekly, because the point of a fresh image is a fresh OS: the build resolves
	# its packages from trixie, trixie-updates and trixie-security at build time
	# with nothing pinned, so a card flashed from last week's image comes up
	# already patched instead of pulling months of updates on first boot. Not on
	# every commit: three emulated builds a push would be about half an hour of
	# runner time each, and nothing about a commit makes last week's OS stale. A
	# tag is the exception, because a tag is somebody saying this one is worth
	# naming.
	#
	# Off the hour on purpose. Scheduled runs queue behind everyone else's, and
	# the ones on :00 wait longest.
	schedule:
	- cron: "17 3 * * 1"
	# A named build, for when a weekly run number is not something to point at:
	# a hardware-validation round, a support thread, a line in a document. The
	# tag becomes the release, so the same three files can be asked for by name
	# a year later instead of by a run number nobody wrote down.
	#
	# The `appliance-` prefix is the load-bearing part, not the `v`.
	# admin/releases.py offers every non-draft release of this repository as an
	# EMS system build and decides by VERSION_PATTERN.fullmatch(tag), so a tag
	# this side of the product must never parse as a version. It does not:
	# the pattern is anchored and appliance-image-v0.1.0 fails it at the first
	# character. tests/test_appliance_image_tag_namespace.py keeps that true.
	push:
	tags:
	- appliance-image-v*
	workflow_dispatch:
	inputs:
	profile:
	description: Which board to build
	type: choice
	options:
	- all
	- rpi3
	- rpi4
	- rpi5
	default: all
	publish:
	description: Publish the finished build as a downloadable prerelease
	type: boolean
	default: true

	permissions:
	contents: read

	# A second run of the same ref would race the first for nothing, but a queued
	# run is not cancelled: a 45-minute build that is nearly done is worth more than
	# the one that would replace it.
	concurrency:
	group: appliance-image-${{ github.ref }}
	cancel-in-progress: false

	jobs:
	plan:
	name: Appliance image plan
	runs-on: ubuntu-latest
	timeout-minutes: 10
	permissions:
	contents: read
	outputs:
	profiles: ${{ steps.pick.outputs.profiles }}

	steps:
	- name: Checkout repository
	uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Resolve the board list from the one table that knows
	id: pick
	env:
	# Never interpolated into the program text: a choice input is
	# constrained by GitHub, and the next input to be added might not be.
	WANTED: ${{ inputs.profile }}
	run: \|
	set -euo pipefail
	python3 - <<'PY' >> "$GITHUB_OUTPUT"
	import json
	import os
	import sys

	sys.path.insert(0, ".")
	from appliance import rpi_image_gen

	known = sorted(rpi_image_gen.HARDWARE_PROFILES)
	wanted = (os.environ.get("WANTED") or "all").strip()
	if wanted in ("", "all"):
	chosen = known
	elif wanted in known:
	chosen = [wanted]
	else:
	raise SystemExit(f"{wanted!r} is not a board this project builds: {known}")
	print("profiles=" + json.dumps(chosen))
	PY
	cat "$GITHUB_OUTPUT"

	manager-package:
	name: Appliance image manager package
	runs-on: ubuntu-latest
	timeout-minutes: 15
	permissions:
	contents: read

	# The .deb the image carries, built on its own. packaging/appliance/build-deb.sh
	# is reproducible from SOURCE_DATE_EPOCH and a pinned compressor, which is
	# what lets it run on a builder nobody attested -- two builds of one commit
	# are the same bytes. Uploaded separately because a Manager fix during a
	# hardware test is then a file copy and a dpkg -i, not a 45-minute image
	# build and a re-flash.
	steps:
	- name: Checkout repository
	uses: actions/checkout@v7
	with:
	fetch-depth: 0
	persist-credentials: false

	- name: Build the arm64 package
	run: \|
	set -euo pipefail
	mkdir -p "${RUNNER_TEMP}/package"
	packaging/appliance/build-deb.sh --output "${RUNNER_TEMP}/package" --arch arm64
	ls -l "${RUNNER_TEMP}/package"

	- name: Report what was built
	run: \|
	set -euo pipefail
	{
	echo "### Appliance Manager package"
	echo
	echo '```'
	cat "${RUNNER_TEMP}"/package/*.deb.sha256
	echo '```'
	} >> "$GITHUB_STEP_SUMMARY"

	- name: Upload the package
	uses: actions/upload-artifact@v7
	with:
	name: appliance-manager-deb
	path: ${{ runner.temp }}/package/
	retention-days: 30
	overwrite: true

	image:
	name: Appliance image build ${{ matrix.profile }}
	needs: plan
	runs-on: ubuntu-latest
	# About 28 minutes per image on the maintainer's machine. A 4-CPU hosted
	# runner running the foreign-architecture stages under emulation is several
	# times that, and the xz of an 8.25 GiB image is single-file work.
	timeout-minutes: 180
	permissions:
	contents: read
	strategy:
	# One board's build says nothing about another board's, so a failure does
	# not cancel the others.
	fail-fast: false
	matrix:
	profile: ${{ fromJSON(needs.plan.outputs.profiles) }}

	steps:
	- name: Reclaim runner disk
	run: \|
	set -euo pipefail
	# The free-space idiom is scripts/lib/workdir.sh's, which cannot be
	# sourced here: this step runs before the checkout. -P and --output
	# are mutually exclusive in coreutils, and a df that refuses to run is
	# a step that fails on its first line.
	avail() { df -PB1 "$1" \| awk 'NR==2 {print $4}'; }
	before="$(avail /)"
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
	/opt/hostedtoolcache/CodeQL /usr/local/share/boost /usr/local/.ghcup \|\| true
	sudo docker image prune --all --force >/dev/null 2>&1 \|\| true
	after="$(avail /)"
	echo "reclaimed $(( (after - before) / 1024 / 1024 )) MiB on /"
	df -h / /mnt \|\| true

	- name: Checkout repository
	uses: actions/checkout@v7
	with:
	# The build refuses to attribute an image to a revision whose commit
	# object it cannot see, and the source bundle is written from the
	# object tree rather than the working directory.
	fetch-depth: 0
	persist-credentials: false

	- name: Choose the filesystem the build writes to
	run: \|
	set -euo pipefail
	# -B points the generator's build directory at $OUTPUT/build-<id>, so
	# --output decides where roughly 15 G of chroot and intermediate
	# images land. On a hosted runner that is not the root filesystem.
	root=""
	for candidate in /mnt "${RUNNER_TEMP}"; do
	sudo mkdir -p "${candidate}/appliance" 2>/dev/null \|\| continue
	sudo chown "$(id -u):$(id -g)" "${candidate}/appliance" 2>/dev/null \|\| continue
	root="${candidate}/appliance"
	break
	done
	[ -n "${root}" ] \|\| { echo "::error::no writable build root on this runner"; exit 1; }

	avail="$(df -PB1 "${root}" \| awk 'NR==2 {print $4}')"
	need=$((30 * 1024 * 1024 * 1024))
	if [ "${avail}" -lt "${need}" ]; then
	echo "::error::${root} has $(( avail / 1024 / 1024 / 1024 )) GiB free; the build needs 30 GiB"
	exit 1
	fi

	mkdir -p "${root}/dist" "${root}/tmp"
	# Every path this job works in, decided here and nowhere else. The
	# generator's location is not job-level env because the runner context
	# does not exist there: a workflow that names it is rejected whole,
	# before a job starts.
	{
	echo "APPLIANCE_OUTPUT=${root}/dist"
	echo "TMPDIR=${root}/tmp"
	echo "EMS_RPI_IMAGE_GEN=${RUNNER_TEMP}/rpi-image-gen"
	} >> "$GITHUB_ENV"
	echo "building into ${root} ($(( avail / 1024 / 1024 / 1024 )) GiB free)"

	- name: Fetch the pinned generator
	run: \|
	set -euo pipefail
	# Verifies the tarball against packaging/appliance/image/rpi-image-gen.lock
	# before extracting it, so this proves the pin as well as fetching it.
	scripts/appliance-fetch-rpi-image-gen.sh --into "${EMS_RPI_IMAGE_GEN}" --form tarball

	- name: Install the generator's declared dependencies
	run: \|
	set -euo pipefail
	export DEBIAN_FRONTEND=noninteractive
	sudo -E apt-get update -qq
	# Upstream's own installer reads its own dependency list, which is the
	# list the builder guest installs too. Its failure is not fatal here:
	# the compatibility probe below is what decides whether this runner can
	# build, and it names every missing binary and package.
	# Invoked through bash rather than as a program: the exec bit is a
	# property of the delivery path, and this one is a tarball.
	if ! sudo -E bash "${EMS_RPI_IMAGE_GEN}/install_deps.sh"; then
	echo "::warning::install_deps.sh did not complete; the probe below decides"
	fi
	# What the builder guest installs beside upstream's list.
	sudo -E apt-get install -y -qq \
	qemu-user-static binfmt-support zstd xz-utils gpgv gdisk

	- name: Make rootless podman work on this runner
	run: \|
	set -euo pipefail
	# rpi-image-gen runs its foreign-architecture stages through rootless
	# podman: its bin/ns helper evals a bash function definition when it is
	# already root and dies before the first layer. Ubuntu 24.04 refuses
	# unprivileged user namespaces through AppArmor, which is exactly what
	# rootless podman and mmdebstrap are.
	if [ -e /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]; then
	sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
	fi
	user="$(id -un)"
	grep -q "^${user}:" /etc/subuid \|\| echo "${user}:100000:65536" \| sudo tee -a /etc/subuid
	grep -q "^${user}:" /etc/subgid \|\| echo "${user}:100000:65536" \| sudo tee -a /etc/subgid
	# A login session is what gives rootless podman its XDG_RUNTIME_DIR,
	# and an Actions step does not have one.
	sudo loginctl enable-linger "${user}" \|\| true
	runtime="/run/user/$(id -u)"
	if [ ! -d "${runtime}" ]; then
	runtime="${RUNNER_TEMP}/runtime"
	mkdir -p "${runtime}"
	chmod 700 "${runtime}"
	fi
	echo "XDG_RUNTIME_DIR=${runtime}" >> "$GITHUB_ENV"
	sudo systemctl restart systemd-binfmt.service \|\| true

	- name: Report what this runner can and cannot do
	run: \|
	set -euo pipefail
	podman system migrate >/dev/null 2>&1 \|\| true
	set +e
	scripts/appliance-check-rpi-image-gen.sh --rpi-image-gen "${EMS_RPI_IMAGE_GEN}"
	status=$?
	set -e
	case "${status}" in
	0) echo "this runner satisfies the generator's contract" ;;
	1) echo "::error::the fetched tree is not the pinned rpi-image-gen contract"; exit 1 ;;
	*) echo "::error::this runner cannot build with the generator; see the missing entries above"
	exit 1 ;;
	esac

	- name: Take the Manager package an operator would be offered
	run: \|
	set -euo pipefail
	# The image should carry the bytes the update path offers, not a
	# second build of the same source that only happens to match. So the
	# build reads the same index the fleet reads and makes the same
	# decision, with the same keyring and the same program.
	#
	# Exit 3 is "no stable release published yet", which is a state this
	# project is actually in and not a failure: the build then makes its
	# own package, and says which of the two happened.
	set +e
	package="$(scripts/appliance-fetch-manager-package.py \
	--index "https://github.com/${GITHUB_REPOSITORY}/releases/download/appliance-manager-index/manager-packages.json" \
	--into "${APPLIANCE_OUTPUT}/manager")"
	status=$?
	set -e
	case "${status}" in
	0) echo "EMS_APPLIANCE_MANAGER_PACKAGE=${package}" >> "$GITHUB_ENV"
	echo "the image will carry the published $(basename "${package}")" ;;
	3) echo "::notice::no stable Manager release is published yet; building one from this checkout" ;;
	*) echo "::error::the published Manager package could not be verified"; exit 1 ;;
	esac

	- name: Capture the builder environment
	run: \|
	set -euo pipefail
	# Recorded honestly rather than completely. A hosted runner has no
	# approved base image and no digest for one, so --base-image-sha512 is
	# left empty: release_inputs.verify_builder_environment then refuses
	# this build for the true reason instead of being handed a digest of
	# something that is not a base image.
	lock_id="ci:github-hosted"
	if [ -r /imagegeneration/imagedata.json ]; then
	lock_id="ci:$(python3 - <<'PY'
	import json

	with open("/imagegeneration/imagedata.json", encoding="utf-8") as handle:
	data = json.load(handle)
	entry = data[0] if isinstance(data, list) and data else {}
	print(f"{entry.get('group', 'unknown')}-{entry.get('version', 'unknown')}".strip())
	PY
	)"
	fi
	scripts/appliance-capture-builder-environment.sh \
	--output "${APPLIANCE_OUTPUT}/builder-environment.json" \
	--base-image-lock-id "${lock_id}" \
	--depends "${EMS_RPI_IMAGE_GEN}/depends"
	cat "${APPLIANCE_OUTPUT}/builder-environment.json"

	- name: Create and verify the source bundle
	run: \|
	set -euo pipefail
	# The source-bundle gate's input: written from the git object tree and
	# verified against it object by object before it is handed over.
	scripts/appliance-create-source-bundle.sh \
	--output "${APPLIANCE_OUTPUT}/source-bundle.tar.gz"

	- name: Run the release gates in builder mode
	env:
	PROFILE: ${{ matrix.profile }}
	run: \|
	set -euo pipefail
	# A tag names the image it builds; every other trigger produces a
	# development build that says so in its own file name. Nothing in the
	# source tree records a version, so this is the only place the answer
	# enters, and scripts/lib/appliance-version.sh gives the same answer
	# to the gates, the build and the finalizer.
	if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then
	VERSION="${GITHUB_REF_NAME#appliance-image-v}"
	export VERSION
	echo "building ${VERSION} from ${GITHUB_REF_NAME}"
	fi
	# --mode builder is the question this runner can answer: can this
	# source, on this builder, produce an image that inspects cleanly? Its
	# best verdict is PASS (builder qualification), which is not a release.
	# Strict is the default, so a required gate that did not run exits 3
	# and is not reported as a pass.
	set +e
	scripts/appliance-release-gates.sh \
	--mode builder \
	--profile "${PROFILE}" \
	--output "${APPLIANCE_OUTPUT}" \
	--rpi-image-gen "${EMS_RPI_IMAGE_GEN}" \
	--builder-environment "${APPLIANCE_OUTPUT}/builder-environment.json" \
	--source-bundle "${APPLIANCE_OUTPUT}/source-bundle.tar.gz" \
	2>&1 \| tee "${APPLIANCE_OUTPUT}/release-gates.log"
	status="${PIPESTATUS[0]}"
	set -e
	exit "${status}"

	- name: Summarise the verdict
	if: always()
	env:
	PROFILE: ${{ matrix.profile }}
	run: \|
	set -euo pipefail
	output="${APPLIANCE_OUTPUT:-}"
	{
	echo "### ${PROFILE}"
	echo
	if [ -n "${output}" ] && [ -r "${output}/release-gates.log" ]; then
	echo '```'
	# The gate list and the verdict, not the whole build log.
	sed -n '/^gate /,$p' "${output}/release-gates.log" \| tail -n 40
	echo '```'
	else
	echo "The gates produced no log; the build did not reach them."
	fi
	echo
	echo "This image is not signable: it was built on a hosted runner,"
	echo "which release policy does not approve as a builder."
	} >> "$GITHUB_STEP_SUMMARY"
	[ -n "${output}" ] && ls -l "${output}" \|\| true

	- name: Upload the image
	if: always() && env.APPLIANCE_OUTPUT != ''
	uses: actions/upload-artifact@v7
	with:
	name: appliance-image-${{ matrix.profile }}
	# The raw .img is 8.25 GiB and mostly empty; Imager and balenaEtcher
	# write the .xz straight to a card, so the operator unpacks nothing.
	path: \|
	${{ env.APPLIANCE_OUTPUT }}/*.img.xz
	${{ env.APPLIANCE_OUTPUT }}/*.img.xz.sha256
	${{ env.APPLIANCE_OUTPUT }}/*.build-authority.json
	${{ env.APPLIANCE_OUTPUT }}/*.build.json
	${{ env.APPLIANCE_OUTPUT }}/builder-environment.json
	if-no-files-found: warn
	retention-days: 30
	overwrite: true

	- name: Upload the gate evidence
	if: always() && env.APPLIANCE_OUTPUT != ''
	uses: actions/upload-artifact@v7
	with:
	name: appliance-gates-${{ matrix.profile }}
	path: \|
	${{ env.APPLIANCE_OUTPUT }}/release-gates.log
	${{ env.APPLIANCE_OUTPUT }}/gates/
	${{ env.APPLIANCE_OUTPUT }}/reports/
	${{ env.APPLIANCE_OUTPUT }}/*.build.log
	if-no-files-found: warn
	retention-days: 30
	overwrite: true

	publish:
	name: Appliance image publish
	needs: [plan, manager-package, image]
	# An Actions artefact expires in 30 days and is reachable only through the
	# run page, by someone signed in with access to this repository. The install
	# page sends an operator to the Releases page instead, for these exact file
	# names, so that is where the files have to be.
	#
	# Only a build every board finished is published. The image job uploads its
	# evidence even when the gates fail -- that is when the logs are worth most
	# -- so the presence of an artefact says nothing here; the gates' verdict
	# does.
	# The `inputs` context exists for workflow_dispatch and workflow_call and is
	# empty for anything else, so a plain `inputs.publish` would make every
	# scheduled run build three images and publish nothing -- as a green skip,
	# with every job reporting success. Declining to publish is a choice a
	# person makes at dispatch time; a schedule has nobody to make it.
	if: >-
	${{ (github.event_name != 'workflow_dispatch' \|\| inputs.publish)
	&& needs.image.result == 'success'
	&& needs['manager-package'].result == 'success' }}
	runs-on: ubuntu-latest
	timeout-minutes: 30
	permissions:
	# The only job in this workflow that may write anything, and all it writes
	# is a release of files the jobs above already built. No secret is read
	# here: the automatic token creates the release, and nothing signs.
	contents: write

	steps:
	- name: Take the release identity these images carry
	uses: actions/checkout@v7

	# The fingerprint a card pins is frozen the moment it is flashed:
	# appliance.conf ships to /usr/share, is not a conffile, and config-seed
	# leaves an existing /etc copy unread. Publishing is therefore the point of
	# no return, and an identity nobody holds the secret half of turns every
	# card it reaches into one that can never install a Manager package again
	# -- not an upgrade, and not the downgrade that is its only recovery.
	# Building stays allowed, because hardware validation needs images.
	- name: Refuse to publish against an identity nobody can sign with
	run: python3 scripts/appliance-check-release-identity.py

	- name: Collect what the build produced
	uses: actions/download-artifact@v7
	with:
	path: ${{ runner.temp }}/collected
	pattern: appliance-*

	- name: Lay out the files a download page needs
	env:
	PROFILES: ${{ needs.plan.outputs.profiles }}
	run: \|
	set -euo pipefail
	collected="${RUNNER_TEMP}/collected"
	release="${RUNNER_TEMP}/release"
	mkdir -p "${release}"

	# The board list comes from the plan job rather than being typed a
	# second time: a release short one board, with every job green, is
	# the failure this workflow was written to prevent.
	boards="$(printf '%s' "${PROFILES}" \
	\| python3 -c 'import json,sys; print(" ".join(json.load(sys.stdin)))')"
	[ -n "${boards}" ] \|\| { echo "::error::the plan job named no board"; exit 1; }

	for board in ${boards}; do
	image="${collected}/appliance-image-${board}"
	[ -d "${image}" ] \|\| { echo "::error::no image artefact for ${board}"; exit 1; }
	# The compressed image and the checksum beside it, under the names
	# docs/user/appliance/install.md tells an operator to look for. The
	# raw .img is 8.25 GiB and never leaves the runner.
	cp "${image}"/.img.xz "${image}"/.img.xz.sha256 "${release}/"
	cp "${image}"/*.build-authority.json "${release}/"
	# One name for three boards, so it needs the board to stay distinct.
	cp "${image}/builder-environment.json" \
	"${release}/builder-environment-${board}.json"

	gates="${collected}/appliance-gates-${board}"
	[ -r "${gates}/release-gates.log" ] \
	\|\| { echo "::error::no gate log for ${board}"; exit 1; }
	( cd "${gates}" && zip -qr "${release}/gate-evidence-${board}.zip" . )
	done

	# Which Manager the images carry, read from their own build records
	# rather than assumed. Since --manager-package the image bakes in the
	# newest published stable Manager and the job above builds one from
	# this checkout: between releases those are the same bytes, and from
	# the first release onwards they are not.
	carried=""
	carried_version=""
	for board in ${boards}; do
	# Chosen by what it contains, not by what it sorts as. When no
	# stable Manager is published the image builds its own .deb, and
	# build-deb.sh writes a second .build.json into the same directory
	# whose name sorts first and which carries no image fields at all --
	# so `ls \| head -1` picked the wrong record in exactly the state
	# this repository is in.
	record="$(python3 -c 'import json,pathlib,sys; print(next((str(p) for p in sorted(pathlib.Path(sys.argv[1]).glob("*.build.json")) if json.loads(p.read_text()).get("appliance_package_sha256")), ""))' "${collected}/appliance-image-${board}")"
	[ -n "${record}" ] \|\| { echo "::error::${board} produced no build record naming a manager package"; exit 1; }
	cp "${record}" "${release}/"
	this="$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(r.get("appliance_package_sha256",""))' "${record}")"
	if [ -n "${carried}" ] && [ "${carried}" != "${this}" ]; then
	echo "::error::the boards do not carry the same Manager package"
	exit 1
	fi
	carried="${this}"
	carried_version="$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(r.get("appliance_package_version",""))' "${record}")"
	done

	package="${collected}/appliance-manager-deb"
	[ -d "${package}" ] \|\| { echo "::error::no manager package"; exit 1; }
	built="$(cut -d' ' -f1 < "$(ls "${package}"/*.deb.sha256 \| head -1)")"

	if [ "${built}" = "${carried}" ]; then
	# The same bytes, so this really is the package these images carry.
	# The .build.json travels with it because
	# appliance-build-manager-manifest.py refuses without it: a published
	# .deb nobody can write a manifest for would have to be rebuilt to be
	# signed, and a rebuild elsewhere records a different builder.
	cp "${package}"/.deb "${package}"/.deb.sha256 \
	"${package}"/*.build.json "${release}/"
	echo "MANAGER_BESIDE=yes" >> "$GITHUB_ENV"
	else
	# Publishing it anyway is how an operator hand-installs an unsigned
	# HEAD build over the released one, under a sentence saying it is
	# what the images carry -- usually under the same file name, with a
	# .sha256 that contradicts the released package's.
	echo "::notice::these images carry Manager ${carried_version}, which has its own release; a checkout build is not put beside them"
	echo "MANAGER_BESIDE=no" >> "$GITHUB_ENV"
	fi
	echo "MANAGER_VERSION=${carried_version}" >> "$GITHUB_ENV"

	# Written outside the directory it describes, so it cannot list itself.
	( cd "${release}" && sha256sum -- * > "${RUNNER_TEMP}/SHA256SUMS" )
	mv "${RUNNER_TEMP}/SHA256SUMS" "${release}/SHA256SUMS"
	ls -l "${release}"

	- name: Say what this build is, and what it is not
	env:
	PROFILES: ${{ needs.plan.outputs.profiles }}
	run: \|
	set -euo pipefail
	release="${RUNNER_TEMP}/release"
	boards="$(printf '%s' "${PROFILES}" \
	\| python3 -c 'import json,sys; print(" ".join(json.load(sys.stdin)))')"
	first="${boards%% *}"
	# The version is read off what was actually built rather than out of
	# the source tree, so the page cannot name a version the files do not.
	version="$(ls "${release}"/-"${first}"-.img.xz \
	\| head -1 \
	\| sed -n "s/.appliance-$.$-${first}-.*\.img\.xz\$/\1/p")"
	[ -n "${version}" ] \|\| { echo "::error::cannot read the version"; exit 1; }
	echo "APPLIANCE_VERSION=${version}" >> "$GITHUB_ENV"

	notes="${RUNNER_TEMP}/release-notes.md"
	{
	echo "Appliance image ${version}, built on a GitHub-hosted runner from"
	echo "\`${GITHUB_SHA}\`, for $(printf '%s' "${PROFILES}" \
	\| python3 -c 'import json,sys; print(", ".join(json.load(sys.stdin)))')."
	echo
	echo "This image is not signable: it was built on a hosted runner,"
	echo "which release policy does not approve as a builder, and a finalizer"
	echo "refuses it with \`builder_environment_untrusted\` by design."
	echo "This is not a release. Cutting one means running the gates in"
	echo "production mode against a signing environment."
	echo
	echo "Flashing verifies no signature, so an unsigned build is enough to"
	echo "write a card with — which is what makes this worth downloading."
	echo "Follow [Flashing the card](https://github.com/${GITHUB_REPOSITORY}/blob/${GITHUB_SHA}/docs/user/appliance/install.md):"
	echo "take the \`.img.xz\` for your board and the \`.img.xz.sha256\` beside"
	echo "it, check the download, and write it with Raspberry Pi Imager."
	echo "\`SHA256SUMS\` covers every file here at once."
	echo
	if [ "${MANAGER_BESIDE}" = yes ]; then
	echo "Also here: the Appliance Manager \`.deb\` these images carry, so a"
	echo "Manager fix on a board under test is a file copy and a \`dpkg -i\`"
	echo "rather than a re-flash."
	else
	echo "These images carry Appliance Manager \`${MANAGER_VERSION}\`, which is"
	echo "published on its own release — take it from there rather than from"
	echo "here, so what you install is the signed package the appliance would"
	echo "have offered you."
	fi
	echo
	echo "Also here: each image's build record and build authority, the builder"
	echo "environment each was recorded with, and the gate logs."
	echo
	echo "The gate runner's own verdict, per board, read back from the log"
	echo "each \`gate-evidence-<board>.zip\` carries rather than asserted here:"
	echo
	echo '```'
	for board in ${boards}; do
	log="${RUNNER_TEMP}/collected/appliance-gates-${board}/release-gates.log"
	# A command substitution's failure is discarded, so an empty read
	# would publish a blank verdict under a sentence promising the
	# verdict was read back from evidence. Refuse instead.
	verdict="$(sed -n 's/^RESULT: //p' "${log}" \| tail -1)"
	[ -n "${verdict}" ] \
	\|\| { echo "::error::${log} states no verdict" >&2; exit 1; }
	printf '%-6s %s\n' "${board}" "${verdict}"
	done
	echo '```'
	} > "${notes}"
	cat "${notes}"

	- name: Publish the download page
	env:
	GH_TOKEN: ${{ github.token }}
	# gh resolves the repository from --repo, GH_REPO or the git remotes of
	# the working directory, in that order, and never from
	# GITHUB_REPOSITORY. This job does not check out, so without this every
	# gh call below dies with "not a git repository" -- at the end of a
	# three-hour build.
	GH_REPO: ${{ github.repository }}
	# A tag push publishes under its own name; every other trigger falls
	# back to the run number. Neither is a version number to
	# admin/releases.py, which lists every non-draft release of this
	# repository as an EMS system-build target and decides eligibility by
	# parsing the tag: a semver-shaped tag here would offer operators a
	# build whose container images do not exist. Both forms keep the
	# `appliance-` prefix that makes VERSION_PATTERN.fullmatch fail.
	TAG: ${{ github.ref_type == 'tag' && github.ref_name \|\| format('appliance-image-ci-{0}', github.run_number) }}
	IS_TAG: ${{ github.ref_type == 'tag' }}
	run: \|
	set -euo pipefail
	state="$(gh release view "${TAG}" --json isDraft --jq .isDraft 2>/dev/null \|\| true)"
	case "${state}" in
	true)
	# A draft left by a failed upload holds no git tag, so removing it
	# costs nothing and is what makes "re-run failed jobs" work: the
	# run number does not change on a retry, so the tag would
	# otherwise be taken by the wreckage of the previous attempt.
	echo "removing the draft left by an earlier attempt"
	gh release delete "${TAG}" --yes ;;
	false)
	echo "::error::${TAG} is already published; delete it or dispatch again"
	exit 1 ;;
	esac

	# Created as a draft and published once, at the end. A release that is
	# still receiving three quarters of a gigabyte is not one anybody
	# should be able to download half of.
	# docs/user/appliance/install.md tells an operator to open the newest
	# entry "whose title starts with Appliance image", because this page
	# also lists EMS releases that carry no image. Both forms therefore
	# open with those two words and name the version after them.
	# "CI build <n>" is true of a weekly run and a lie about a tagged one;
	# unsigned is true of both and stays in both, because a hosted runner
	# is refused at signing time by design.
	if [ "${IS_TAG}" = "true" ]; then
	title="Appliance image ${APPLIANCE_VERSION} (${TAG}, unsigned)"
	else
	title="Appliance image ${APPLIANCE_VERSION} (CI build ${GITHUB_RUN_NUMBER}, unsigned)"
	fi

	gh release create "${TAG}" \
	--draft \
	--target "${GITHUB_SHA}" \
	--title "${title}" \
	--notes-file "${RUNNER_TEMP}/release-notes.md" \
	"${RUNNER_TEMP}"/release/*

	# Prerelease, and still load-bearing after the fleet's index moved
	# to its own tag. /releases/latest resolves to the newest release
	# that is neither draft nor prerelease, and it is what the repository
	# sidebar advertises and what anything asking this repository for
	# "the latest release" is handed. An ordinary release here would put
	# an unsigned appliance image in that position every week, in front
	# of the EMS version it is not, on the very page
	# docs/user/appliance/install.md sends operators to.
	gh release edit "${TAG}" --draft=false --prerelease

	url="$(gh release view "${TAG}" --json url --jq .url)"
	{
	echo "### Published"
	echo
	echo "[${TAG}](${url}) — unsigned prerelease, one image per board."
	} >> "$GITHUB_STEP_SUMMARY"

	- name: Keep the fixed download link pointing at this build
	env:
	GH_TOKEN: ${{ github.token }}
	GH_REPO: ${{ github.repository }}
	TAG: ${{ github.ref_type == 'tag' && github.ref_name \|\| format('appliance-image-ci-{0}', github.run_number) }}
	POINTER_TAG: appliance-image-latest
	run: \|
	set -euo pipefail
	# Where somebody is sent who has no version in mind. The releases above
	# are the record — three files that can still be asked for by name a
	# year later — and this is the signpost to the newest of them.
	#
	# It exists because the Releases page mixes two products and its
	# "Latest" badge deliberately names an EMS release that carries no
	# image. Without one fixed address, every sentence we write about
	# downloading is a search instruction: open the list, skip the entries
	# that are not images, find the newest one that is. That is the step
	# someone installing their first appliance gets wrong.
	#
	# Created once and then only rewritten, exactly like
	# appliance-manager-index: the tag never moves, so a link printed in a
	# document — or in a forum post two years old — stays true.
	#
	# It deliberately carries no image of its own. Copying three quarters
	# of a gigabyte here every week would buy a download URL whose file
	# name has lost the version it identifies itself by, and a window
	# during which half the assets are last week's while the checksums
	# beside them are this week's.
	#
	# Its title deliberately does not open with "Appliance image". The
	# install guide tells an operator to find the newest entry that does,
	# and this entry has no image to hand them.
	pointer_title="Download the appliance image"
	pointer="${RUNNER_TEMP}/appliance-image-latest.json"
	notes="${RUNNER_TEMP}/pointer-notes.md"

	# Read back off the release rather than rebuilt from the names this
	# job used, so the pointer can only ever name files that are actually
	# published under it.
	gh release view "${TAG}" --json tagName,url,assets --jq \
	'{tag: .tagName, url: .url,
	images: [.assets[] \| select(.name \| endswith(".img.xz")) \| {name, url}]}' \
	> "${pointer}"
	count="$(jq '.images \| length' "${pointer}")"
	[ "${count}" -gt 0 ] \|\| {
	echo "::error::${TAG} publishes no .img.xz, so there is nothing to point at"
	exit 1
	}

	# Read out of the pointer rather than reused from the step above: a
	# `run:` block is its own shell, so a variable set in the previous one
	# is unset here and `set -u` would end a thirty-minute build on the
	# last line of it.
	release_url="$(jq -r .url "${pointer}")"
	{
	echo "The newest appliance image build: ${APPLIANCE_VERSION}, published as"
	echo "[\`${TAG}\`](${release_url})."
	echo
	echo "One file per board — they are not interchangeable:"
	echo
	jq -r '.images[] \| "- [`\(.name)`](\(.url))"' "${pointer}"
	echo
	echo "Download the \`.img.xz.sha256\` beside your file as well, and check it"
	echo "before writing the card — see"
	echo "[Flashing the card](https://github.com/${GH_REPO}/blob/main/docs/user/appliance/install.md)."
	echo
	echo "Unsigned, and that is by design: an image is only signable on an approved"
	echo "builder, which a hosted runner is not. Writing a card verifies no signature"
	echo "either way, so the checksum is the check that exists."
	echo
	echo "This tag never moves. The images keep their version in their file names and"
	echo "live in the release above, which is never rewritten."
	} > "${notes}"
	cat "${notes}"

	if gh release view "${POINTER_TAG}" >/dev/null 2>&1; then
	gh release edit "${POINTER_TAG}" --title "${pointer_title}" --notes-file "${notes}"
	else
	gh release create "${POINTER_TAG}" \
	--prerelease \
	--target "${GITHUB_SHA}" \
	--title "${pointer_title}" \
	--notes-file "${notes}"
	fi
	gh release upload "${POINTER_TAG}" "${pointer}" --clobber

	pointer_url="$(gh release view "${POINTER_TAG}" --json url --jq .url)"
	{
	echo
	echo "Fixed download link: [${POINTER_TAG}](${pointer_url})"
	} >> "$GITHUB_STEP_SUMMARY"

Navigation Menu

Merge pull request #159 from basecubedev/fix/appliance-home-identity-… #5

Workflow file

Merge pull request #159 from basecubedev/fix/appliance-home-identity-… #5

Uh oh!

Workflow file for this run