Skip to content

Commit 9cd12bc

Browse files
mlatazamlataza
committed
test: [sc-106110] Cover the capped and jittered update retries in the integration workflow
The cap and the jitter are unit tested against the schedule function, which cannot show that a real agent facing an unavailable release endpoint now retries on bounded, spread-out slots instead of a synchronized schedule that can overflow into a busy-spin. Add a scenario that drives the retry loop on all three platforms against an endpoint that fails and recovers on demand. A GitHub outage is not something CI can arrange, and an unroutable address produces connection errors rather than the HTTP failure the ticket describes, so the scenario needs a stub the agent will actually query. The release endpoint is compiled in, so it gets a build-time-gated seam: releaseUrlOverrideFileStr names a file in the org's data directory whose contents replace the endpoint, and it is injected only into the integration binary. A released build leaves it empty and never looks for the file at all, so a shipped agent's update source stays fixed at build time and cannot be redirected by dropping a file on an endpoint. Unit tests pin both halves: the released build ignoring a planted override, and the integration build honoring a valid one while falling back to the compiled-in endpoint for a missing, empty or whitespace-only file - a fixture that failed to write the file leaves the agent updating normally rather than silently not updating at all. The fixture, test/stubrelease, answers over plain HTTP on loopback, so unlike the stub broker nothing has to be installed into the host trust store. It fails every request with 503 until its mode file flips, which it re-reads per request, so the endpoint can recover in the middle of a retry sequence. It logs every arrival with millisecond precision, which is the request-spacing record the ticket asks for, and serves the running agent's own version as tag_name so a recovered check ends at "No updates available" - the recovery is what is under test, not the installer. The scenario points the agent at the failing stub, installs the integration binary with auto-updates on, waits for three retries, flips the endpoint mid-sequence and asserts the update succeeds on a retry, then asserts the schedule itself: it prints an attempt/backoff/elapsed table and checks every slot is strictly positive, within the cap, jittered, growing, and never served back to back. Distinct backoff values alone would be weak evidence - the doubling produces those on its own until it reaches the ceiling - so the slots that do reach the ceiling must differ from each other, which is what an unjittered schedule cannot do. It then fails the endpoint again and asserts a service stop issued mid-backoff is honored in under 20s rather than waiting out the remaining budget. Every restore step runs under always(), so a failed assertion cannot leave the rest of the job running an agent pointed at a stub endpoint or leave the service stopped. The integration build's base backoff drops from 10s to 2s. With the 30s check interval the cap is a quarter of it (7.5s), so a 10s base was clamped on every slot and the doubling was unobservable; at 2s the schedule climbs 2s, 4s, then flattens at the cap, and both halves of the fix show up in one sequence. Verified locally as far as a macOS host allows: the fixture serves 503 and the release payload correctly, driving the real AutoUpdateRunner against it produced 1.87s, 3.48s, 7.5s, 6.29s, 7.49s, 6.41s - capped and jittered - and a mid-sequence flip logged "Update succeeded on retry: attempt=3". The analysis script passes on that real log and fails correctly on synthetic pre-fix logs: uncapped doubling, negative durations arriving back to back, and a capped but unjittered schedule. Every new PowerShell block parses under pwsh and actionlint is clean. The actions themselves - service managers, the WMI launch, elevation - can only run on the runners.
1 parent f580066 commit 9cd12bc

9 files changed

Lines changed: 1075 additions & 6 deletions

File tree

‎.github/actions/set-release-url/action.yml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
name: Set agent release url
2+
description: >
3+
Point the installed integration-test agent at a different release endpoint by
4+
writing the override file its build honors (sc-106110), or remove that file to
5+
put it back on the real GitHub releases API. Used to drive the auto-update
6+
retry schedule against the local stub release endpoint.
7+
8+
Only the integration build looks for this file at all - released builds compile
9+
the endpoint in - so this is inert against a release binary. The agent reads it
10+
when a service cycle constructs its updater, so the change takes effect on the
11+
next service start. See fixture.ps1 for details.
12+
13+
inputs:
14+
mode:
15+
description: "set or clear"
16+
required: true
17+
config_dir:
18+
description: "Directory containing the per-org agent data directory (without org id suffix)"
19+
required: true
20+
org_id:
21+
description: "Organization id whose agent should be pointed at the endpoint"
22+
required: true
23+
url:
24+
description: "Release endpoint URL to write (required when mode is set)"
25+
required: false
26+
default: ""
27+
file_name:
28+
description: "Override file name, matching releaseUrlOverrideFileStr in the integration build"
29+
required: false
30+
default: "release_url_override"
31+
32+
runs:
33+
using: composite
34+
steps:
35+
- name: Set release url (Unix)
36+
if: runner.os != 'Windows'
37+
shell: bash
38+
env:
39+
CONFIG_DIR: ${{ inputs.config_dir }}
40+
ORG_ID: ${{ inputs.org_id }}
41+
MODE: ${{ inputs.mode }}
42+
URL: ${{ inputs.url }}
43+
FILE_NAME: ${{ inputs.file_name }}
44+
run: |
45+
set -euo pipefail
46+
# Resolve pwsh up front: sudo resets PATH via secure_path on some runners.
47+
pwsh_path="$(command -v pwsh)"
48+
sudo "$pwsh_path" -NoProfile -File "$GITHUB_ACTION_PATH/fixture.ps1" \
49+
-OverrideFile "$CONFIG_DIR/$ORG_ID/$FILE_NAME" \
50+
-Mode "$MODE" \
51+
-Url "$URL"
52+
53+
- name: Set release url (Windows)
54+
if: runner.os == 'Windows'
55+
shell: pwsh
56+
env:
57+
CONFIG_DIR: ${{ inputs.config_dir }}
58+
ORG_ID: ${{ inputs.org_id }}
59+
MODE: ${{ inputs.mode }}
60+
URL: ${{ inputs.url }}
61+
FILE_NAME: ${{ inputs.file_name }}
62+
run: |
63+
& "$env:GITHUB_ACTION_PATH/fixture.ps1" `
64+
-OverrideFile (Join-Path (Join-Path $env:CONFIG_DIR $env:ORG_ID) $env:FILE_NAME) `
65+
-Mode $env:MODE `
66+
-Url $env:URL

‎.github/actions/set-release-url/fixture.ps1‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/usr/bin/env pwsh
2+
#Requires -Version 7
3+
4+
# Writes (or removes) the release-url override file the integration build reads
5+
# to decide which endpoint the auto-updater queries (sc-106110; see
6+
# agent.ResolveLatestReleaseUrl).
7+
#
8+
# The override lives beside the agent's config in the org's data directory
9+
# because that directory is what the service account can already read on all
10+
# three platforms, whatever account the service runs as. The file is read when a
11+
# service cycle constructs its updater, so writing it takes effect on the next
12+
# service start rather than mid-run.
13+
#
14+
# The directory belongs to the service account (root / SYSTEM), so this runs
15+
# elevated; the wrapping action supplies the elevation per platform.
16+
17+
param(
18+
[Parameter(Mandatory = $true)][string]$OverrideFile,
19+
[Parameter(Mandatory = $true)][ValidateSet('set', 'clear')][string]$Mode,
20+
[string]$Url
21+
)
22+
23+
$ErrorActionPreference = 'Stop'
24+
25+
if ($Mode -eq 'clear') {
26+
# Tolerates a missing file so an always() cleanup can run even when the
27+
# scenario failed before writing one.
28+
Remove-Item -Path $OverrideFile -Force -ErrorAction SilentlyContinue
29+
Write-Output "release url override cleared ($OverrideFile)"
30+
exit 0
31+
}
32+
33+
if ([string]::IsNullOrWhiteSpace($Url)) {
34+
Write-Error "a url is required when setting the release url override"
35+
exit 1
36+
}
37+
38+
$directory = Split-Path -Parent $OverrideFile
39+
if (-not (Test-Path $directory)) {
40+
Write-Error "agent data directory not found at $directory"
41+
exit 1
42+
}
43+
44+
Set-Content -Path $OverrideFile -Value $Url -NoNewline
45+
# The service may run as an account other than the one that wrote this, so the
46+
# file has to stay readable by it; the data directory's own permissions decide
47+
# who can get this far.
48+
if ($IsLinux -or $IsMacOS) {
49+
chmod 0644 $OverrideFile
50+
}
51+
Write-Output "release url override set to $Url ($OverrideFile)"

‎.github/actions/stub-release/action.yml‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
name: Stub release endpoint
2+
description: >
3+
Run the test/stubrelease fixture on the loopback interface as a stand-in for
4+
the GitHub releases API, so the agent's auto-update check can be made to fail
5+
and recover on demand (sc-106110). A real GitHub outage or rate-limit response
6+
is not something CI can arrange, and an unroutable address produces connection
7+
errors rather than the HTTP failure the ticket describes.
8+
9+
The endpoint answers over plain HTTP on 127.0.0.1, so - unlike the stub broker -
10+
nothing has to be installed into the host trust store. The agent is pointed at
11+
it with the set-release-url action, which writes the override file the
12+
integration build honors.
13+
14+
On Windows the fixture must be launched detached (via WMI), because a child
15+
started with Start-Process -NoNewWindow shares the launching step's console and
16+
job object and is reaped when that step ends - the lesson the stub broker
17+
learned the hard way.
18+
19+
mode=start builds and launches the endpoint in failing mode and verifies it
20+
answers before any agent is pointed at it, so a broken fixture fails here
21+
rather than as a mystifying assertion failure a minute later. mode=ok /
22+
mode=fail flip the behavior per request without restarting anything, so the
23+
endpoint can recover in the middle of a retry sequence. mode=stop tears it down
24+
and reports its request log (the arrival-time record for the scenario); it is
25+
idempotent and tolerant of missing state, so an always() cleanup step can run
26+
it even if start never completed.
27+
28+
inputs:
29+
mode:
30+
description: "start, ok, fail, or stop"
31+
required: true
32+
host:
33+
description: "Loopback address the endpoint answers on"
34+
required: false
35+
default: "127.0.0.1"
36+
port:
37+
description: "Port to listen on"
38+
required: false
39+
default: "8765"
40+
tag:
41+
description: >
42+
tag_name served in the release payload. Defaults to the integration build's
43+
own version so a successful check ends at "No updates available" and the
44+
recovery is observed without the agent downloading and executing anything.
45+
required: false
46+
default: "v0.0.0-it"
47+
48+
outputs:
49+
url:
50+
description: "Release endpoint URL to point the agent at"
51+
value: http://${{ inputs.host }}:${{ inputs.port }}/releases/latest
52+
log_file:
53+
description: "Path the endpoint's request log is written to"
54+
value: ${{ steps.paths.outputs.log_file }}
55+
56+
runs:
57+
using: composite
58+
steps:
59+
- name: Resolve working paths
60+
id: paths
61+
shell: pwsh
62+
env:
63+
RUNNER_TEMP_DIR: ${{ runner.temp }}
64+
run: |
65+
$workdir = Join-Path $env:RUNNER_TEMP_DIR 'stub-release'
66+
New-Item -ItemType Directory -Force -Path $workdir | Out-Null
67+
"workdir=$workdir" >> $env:GITHUB_OUTPUT
68+
"log_file=$(Join-Path $workdir 'release.log')" >> $env:GITHUB_OUTPUT
69+
"mode_file=$(Join-Path $workdir 'mode')" >> $env:GITHUB_OUTPUT
70+
"pid_file=$(Join-Path $workdir 'release.pid')" >> $env:GITHUB_OUTPUT
71+
72+
- name: Setup go
73+
if: inputs.mode == 'start'
74+
uses: actions/setup-go@v6
75+
with:
76+
go-version: "1.25.0"
77+
cache: true
78+
79+
# Started in failing mode so a harness that forgets to set a mode reproduces
80+
# the failing endpoint the scenario is built around rather than passing.
81+
- name: Launch endpoint (Unix)
82+
if: inputs.mode == 'start' && runner.os != 'Windows'
83+
shell: bash
84+
env:
85+
WORKDIR: ${{ steps.paths.outputs.workdir }}
86+
MODE_FILE: ${{ steps.paths.outputs.mode_file }}
87+
LOG_FILE: ${{ steps.paths.outputs.log_file }}
88+
PID_FILE: ${{ steps.paths.outputs.pid_file }}
89+
HOST: ${{ inputs.host }}
90+
PORT: ${{ inputs.port }}
91+
TAG: ${{ inputs.tag }}
92+
run: |
93+
set -euo pipefail
94+
printf 'fail' > "$MODE_FILE"
95+
rm -f "$LOG_FILE"
96+
# The fixture is behind the `integration` build tag; see its package doc.
97+
go build -tags integration -o "$WORKDIR/stubrelease" ./test/stubrelease
98+
# It binds a high loopback port and reads only files it owns, so it runs
99+
# unprivileged; the agent reaches it over loopback whatever account the
100+
# service runs as. It writes its own log (-log) rather than having the
101+
# shell redirect it, so both platforms capture output the same way.
102+
nohup "$WORKDIR/stubrelease" \
103+
-listen "$HOST:$PORT" \
104+
-mode-file "$MODE_FILE" \
105+
-tag "$TAG" \
106+
-log "$LOG_FILE" > /dev/null 2>&1 &
107+
echo $! > "$PID_FILE"
108+
echo "Stub release endpoint started (pid $(cat "$PID_FILE"))"
109+
110+
- name: Launch endpoint (Windows)
111+
if: inputs.mode == 'start' && runner.os == 'Windows'
112+
shell: pwsh
113+
env:
114+
WORKDIR: ${{ steps.paths.outputs.workdir }}
115+
MODE_FILE: ${{ steps.paths.outputs.mode_file }}
116+
LOG_FILE: ${{ steps.paths.outputs.log_file }}
117+
PID_FILE: ${{ steps.paths.outputs.pid_file }}
118+
HOST: ${{ inputs.host }}
119+
PORT: ${{ inputs.port }}
120+
TAG: ${{ inputs.tag }}
121+
run: |
122+
$ErrorActionPreference = 'Stop'
123+
Set-Content -Path $env:MODE_FILE -Value 'fail' -NoNewline
124+
Remove-Item -Path $env:LOG_FILE -ErrorAction SilentlyContinue
125+
126+
$exe = Join-Path $env:WORKDIR 'stubrelease.exe'
127+
# The fixture is behind the `integration` build tag; see its package doc.
128+
go build -tags integration -o $exe ./test/stubrelease
129+
if ($LASTEXITCODE -ne 0) { throw "failed to build stub release endpoint" }
130+
131+
# Launched through WMI rather than Start-Process: a process started with
132+
# Start-Process -NoNewWindow shares this step's console and job object,
133+
# so the runner reaps it the moment the step ends - which surfaces later
134+
# as "the target machine actively refused it". Win32_Process.Create
135+
# parents it to WmiPrvSE instead, outside both. WMI offers no stream
136+
# redirection, which is why the fixture takes -log and writes its own.
137+
$arguments = @(
138+
"`"$exe`"",
139+
"-listen", "`"$($env:HOST):$($env:PORT)`"",
140+
"-mode-file", "`"$($env:MODE_FILE)`"",
141+
"-tag", $env:TAG,
142+
"-log", "`"$($env:LOG_FILE)`""
143+
) -join ' '
144+
$created = Invoke-CimMethod -ClassName Win32_Process -MethodName Create `
145+
-Arguments @{ CommandLine = $arguments }
146+
if ($created.ReturnValue -ne 0) {
147+
throw "Win32_Process.Create failed with return value $($created.ReturnValue)"
148+
}
149+
Set-Content -Path $env:PID_FILE -Value $created.ProcessId
150+
Write-Output "Stub release endpoint started (pid $($created.ProcessId))"
151+
152+
# Proves the endpoint is actually answering before an agent is pointed at it.
153+
# Without this a reaped or unstarted fixture is indistinguishable from the
154+
# agent failing to retry, which is what the scenario's assertions measure.
155+
- name: Verify the endpoint is reachable
156+
if: inputs.mode == 'start'
157+
shell: pwsh
158+
env:
159+
HOST: ${{ inputs.host }}
160+
PORT: ${{ inputs.port }}
161+
LOG_FILE: ${{ steps.paths.outputs.log_file }}
162+
PID_FILE: ${{ steps.paths.outputs.pid_file }}
163+
run: |
164+
$url = "http://$($env:HOST):$($env:PORT)/releases/latest"
165+
for ($i = 1; $i -le 15; $i++) {
166+
try {
167+
# In failing mode the endpoint answers 503, which Invoke-WebRequest
168+
# raises on - a raised HTTP status still proves it is listening, so
169+
# only a transport-level failure is treated as not-yet-up.
170+
$response = Invoke-WebRequest -Uri $url -Method Get -SkipHttpErrorCheck -TimeoutSec 5
171+
Write-Output "Stub release endpoint answered $($response.StatusCode) after $i attempt(s)"
172+
exit 0
173+
} catch {
174+
$lastError = $_
175+
Start-Sleep -Seconds 1
176+
}
177+
}
178+
179+
$alive = $false
180+
if (Test-Path $env:PID_FILE) {
181+
$stubPid = (Get-Content $env:PID_FILE).Trim()
182+
$alive = $null -ne (Get-Process -Id $stubPid -ErrorAction SilentlyContinue)
183+
Write-Output "Endpoint process $stubPid alive: $alive"
184+
}
185+
Write-Output "Endpoint log follows:"
186+
if (Test-Path $env:LOG_FILE) { Get-Content $env:LOG_FILE -ErrorAction SilentlyContinue }
187+
Write-Error "Stub release endpoint not reachable (process alive: $alive): $lastError"
188+
exit 1
189+
190+
- name: Set endpoint mode
191+
if: inputs.mode == 'ok' || inputs.mode == 'fail'
192+
shell: pwsh
193+
env:
194+
MODE: ${{ inputs.mode }}
195+
MODE_FILE: ${{ steps.paths.outputs.mode_file }}
196+
run: |
197+
# Read per request, so the change applies to the very next retry without
198+
# restarting the endpoint or the agent.
199+
Set-Content -Path $env:MODE_FILE -Value $env:MODE -NoNewline
200+
Write-Output "Stub release endpoint mode set to $($env:MODE)"
201+
202+
# The request log is the arrival-time record: it shows retries arriving
203+
# spread out rather than in the same millisecond, which is the fleet-facing
204+
# half of what sc-106110 fixed.
205+
- name: Report endpoint request log
206+
if: inputs.mode == 'stop'
207+
shell: pwsh
208+
env:
209+
LOG_FILE: ${{ steps.paths.outputs.log_file }}
210+
run: |
211+
if (Test-Path $env:LOG_FILE) {
212+
Write-Output "---- $($env:LOG_FILE) ----"
213+
Get-Content $env:LOG_FILE -ErrorAction SilentlyContinue
214+
}
215+
216+
- name: Tear down endpoint (Unix)
217+
if: inputs.mode == 'stop' && runner.os != 'Windows'
218+
shell: bash
219+
env:
220+
PID_FILE: ${{ steps.paths.outputs.pid_file }}
221+
run: |
222+
# Tolerates missing state so this can run from an always() cleanup even
223+
# when start never got far.
224+
if [ -f "$PID_FILE" ]; then
225+
kill "$(cat "$PID_FILE")" 2>/dev/null || true
226+
rm -f "$PID_FILE"
227+
fi
228+
echo "Stub release endpoint torn down"
229+
230+
- name: Tear down endpoint (Windows)
231+
if: inputs.mode == 'stop' && runner.os == 'Windows'
232+
shell: pwsh
233+
env:
234+
PID_FILE: ${{ steps.paths.outputs.pid_file }}
235+
run: |
236+
if (Test-Path $env:PID_FILE) {
237+
$stubPid = Get-Content $env:PID_FILE
238+
Stop-Process -Id $stubPid -Force -ErrorAction SilentlyContinue
239+
Remove-Item $env:PID_FILE -ErrorAction SilentlyContinue
240+
}
241+
Write-Output "Stub release endpoint torn down"

0 commit comments

Comments
 (0)