Add apt-retry wrapper for transient apt mirror failures

[leighmcculloch]

What

Add a bash wrapper script that retries apt commands with decorrelated jitter backoff (up to 5 attempts). Apply it to all apt-get update and install commands in the Dockerfile builder stages and dependencies script.

Why

@sisuresh raised today that he'd seen intermittent failures relating to apt installs.

Docker builds intermittently fail due to transient apt mirror issues. Analysis of recent failed runs found the following apt-related failures:

Run	Stage	Error Type	Details
20799747898	apt-get install	404 Not Found	libglib2.0-0t64_2.80.0-6ubuntu3.5_arm64.deb
20793261714	apt-get install	404 Not Found	libxslt1.1_1.1.39-0exp1ubuntu0.24.04.3_amd64.deb
19283455787	apt-get update	Size mismatch	Packages.gz - Mirror sync in progress?
19276873082	apt-get update	Size mismatch	Packages.gz - Mirror sync in progress?

There are two failure patterns:

1. 404 during install - The package index is stale. apt-get update ran and cached version info, but by the time install runs, the mirror has newer versions and old .deb files are gone.
2. Size mismatch during update - The mirror is actively syncing while apt-get update is running.

Both are transient and benefit from retries. The 404 errors during install require re-running apt-get update before retrying install, which is why this PR wraps the entire apt-get update && apt-get install sequence together rather than wrapping each command separately.

Close #881

[Copilot]

Pull request overview

This PR adds a bash wrapper script apt-retry that implements retry logic with decorrelated jitter backoff to handle transient apt mirror failures during Docker builds. The wrapper is applied to all apt-get commands across multiple Dockerfile builder stages and the dependencies script.

Key Changes:

• New apt-retry bash script with decorrelated jitter backoff (up to 5 attempts)
• Updated Dockerfile to use apt-retry wrapper for all apt-get operations in 5 builder stages
• Updated dependencies script to use apt-retry for PostgreSQL and package installations

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File	Description
apt-retry	New bash script implementing retry logic with decorrelated jitter backoff for apt commands
Dockerfile	Copies apt-retry to /usr/local/bin/ in 5 builder stages and wraps all apt-get update/install commands
dependencies	Wraps apt-get update and install commands with apt-retry to handle transient failures

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.