fix(ic-os): serialize podman's SHM lock creation to stop flaky exit-125 builds

ic-os container builds intermittently die with:

  Error: failed to get new shm lock manager: failed to create 2048 locks in
  /libpod_rootless_lock_1001: file exists

Podman keeps its container lock manager in a shared memory segment (one per
uid when rootless, a single unsuffixed one as root). It opens that segment if
it exists and otherwise creates it O_EXCL, with nothing serializing the gap
between those two steps. ic-os actions run concurrently and unsandboxed
(--strategy_regexp=ic-os[:/].*=local), so on a machine where the segment does
not exist yet -- a fresh CI container -- the first burst of actions all find
it missing and race to create it, and the losers exit 125.

That is why this only ever bites early in a job: measured here, 832
concurrent invocations with the segment already present produced 0 failures,
while deleting it and running 16 concurrently reproduced the error.

Materialize the segment exactly once under a lock of our own, before the
build starts. Once it exists podman only ever opens it, so every later action
pays a single `test -e` -- no lock, no extra podman, and the builds
themselves are never serialized. Also applied to build-bootloader-tree.sh,
which drives podman outside the wrapper and can be part of the same first
burst; two independent pre-warms would defeat each other.

Measured with 5 rounds of 16 concurrent invocations from a cold segment,
counting actions that reach podman while the segment is still absent (the
race-exposed state):

              before    after
  rootless     80/80     0/80
  root         80/80     0/80

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: basvandijk

ic-os/bootloader/build-bootloader-tree.sh

@@ -25,6 +25,24 @@ done
 
 TMP_DIR=$(mktemp -d --tmpdir="/tmp/containers" build-image-XXXXXXXXXXXX)
 
+# Materialize podman's SHM lock segment under a lock, exactly as
+# toolchains/sysimage/proc_wrapper.sh does -- see the long comment there. This
+# script drives podman outside that wrapper, so without this it can be part of
+# the same first burst of ic-os actions that race to create the segment.
+if [ "$EUID" -eq 0 ]; then
+    PODMAN_LOCK_SEGMENT="/dev/shm/libpod_lock"
+else
+    PODMAN_LOCK_SEGMENT="/dev/shm/libpod_rootless_lock_$EUID"
+fi
+if [ -w /dev/shm ] && [ ! -e "$PODMAN_LOCK_SEGMENT" ]; then
+    (
+        flock 9
+        if [ ! -e "$PODMAN_LOCK_SEGMENT" ]; then
+            podman --root "${TMP_DIR}/root" --runroot "${TMP_DIR}/runroot" info >/dev/null 2>&1 || true
+        fi
+    ) 9>"/dev/shm/icos-podman-lock-init.$EUID.lck"
+fi
+
 BASE_IMAGE="$(cat ${BASE_IMAGE_FILE})"
 
 podman --root "${TMP_DIR}/root" --runroot "${TMP_DIR}/runroot" build --iidfile "${TMP_DIR}/iidfile" - <<<"

toolchains/sysimage/proc_wrapper.sh

@@ -12,6 +12,43 @@ set -euo pipefail
 mkdir -p /tmp/containers
 podman_storage_dir=$(mktemp -d --tmpdir="/tmp/containers" "icosbuildXXXX")
 
+# Podman keeps its container lock manager in a POSIX shared memory segment: one
+# per uid when rootless, and a single unsuffixed one when running as root. It
+# opens that segment if it exists and otherwise creates it O_EXCL -- with nothing
+# serializing the gap between those two steps. ic-os actions run concurrently and
+# unsandboxed, so on a machine where the segment does not exist yet (a fresh CI
+# container) the first burst of actions all find it missing and race to create it.
+# The losers die:
+#   Error: failed to get new shm lock manager: failed to create 2048 locks in
+#   /libpod_rootless_lock_<uid>: file exists
+# Close that window by materializing the segment exactly once, under a lock of
+# our own, before the build starts. Once it exists podman only ever opens it, so
+# every later action pays a single `test -e`: no lock, no extra podman, and the
+# builds themselves are never serialized.
+#
+# Note that root is NOT exempt: rootful podman keeps the same kind of segment,
+# just under a different name (/run/libpod is its tmp dir, not its lock), and
+# ci/container/container-run.sh runs the build as root whenever the host uid has
+# no matching container user. So pick the name by euid rather than skipping.
+if [ "$EUID" -eq 0 ]; then
+    podman_lock_segment="/dev/shm/libpod_lock"
+else
+    podman_lock_segment="/dev/shm/libpod_rootless_lock_$EUID"
+fi
+if [ -w /dev/shm ] && [ ! -e "$podman_lock_segment" ]; then
+    (
+        flock 9
+        # Re-check under the lock: whoever got here first already created it.
+        if [ ! -e "$podman_lock_segment" ]; then
+            # Any podman command initializes the lock manager; `info` is the
+            # cheapest. A failure here is not fatal -- we are only pre-warming,
+            # and the build below reports real podman problems itself.
+            podman --root "$podman_storage_dir/root" --runroot "$podman_storage_dir/runroot" \
+                info >/dev/null 2>&1 || true
+        fi
+    ) 9>"/dev/shm/icos-podman-lock-init.$EUID.lck"
+fi
+
 tmpdir=$(mktemp -d --tmpdir "icosbuildXXXX")
 # podman runs rootless and writes files under the storage dir owned by
 # mapped subordinate uids; the calling user can't `rm` them directly.