Skip to content

Commit 0bcc41b

Browse files
adamtasteslikegoodadamtasteslikegood
authored
chore(staging): release-train staging gate + CI auto-deploy [KAN-244] (#3423)
## Summary [KAN-244] Staging environment polish — three acceptance criteria for Sprint 8 S4: 1. **Release-train integration**: `train-run.sh --verify-only` now runs a staging health gate before proceeding to production verification. Checks that staging returns 200 and reports `environment=staging` (guards against STAGING_URL accidentally pointing at production). Default URL hardcoded to the known Cloud Run service so the step needs no gcloud credentials. 2. **CI image rebuild**: New `staging-deploy.yml` workflow triggers on push to `dev` (paths-ignore for docs/specs/md). Builds both Docker images, pushes to prod Artifact Registry with `staging-<SHA>` tags, and swaps the staging Cloud Run services. Post-deploy smoke test with cold-start retries, environment=staging verification, and DB-backed `/browse` check. 3. **RUNBOOK documentation**: Step 9 now documents the staging health gate and the `STAGING_URL` override. ### Deployment prerequisites - **GitHub WIF configuration**: The CI workflow requires `GCP_WORKLOAD_IDENTITY_PROVIDER` and `GCP_SERVICE_ACCOUNT` secrets. The CI service account needs push access to the production Artifact Registry and permission to update the staging Cloud Run services and migration Job. - **Cross-project image pulls**: The staging Cloud Run runtime identities need `roles/artifactregistry.reader` on the production Artifact Registry project. - **Staging migration Job**: `flask-staging-migrate` must exist in the staging project with the staging database configuration. The workflow updates it to the just-built Flask image, executes it synchronously before service rollout, and fails closed if the Job is missing or the migration fails. - **Smoke validation**: `/browse` is a fatal DB-backed Flask probe; a green deploy therefore requires Express, Flask, and the staging database to respond successfully. ## Test plan - [x] `bash -n scripts/release/train-run.sh` — shell syntax OK - [x] `python3 -c "import yaml; yaml.safe_load(...)"` — YAML syntax OK - [x] `npx prettier --check scripts/release/RUNBOOK.md` — format OK - [x] **Negative test**: `STAGING_URL=https://www.tasteslikegood.org ./scripts/release/train-run.sh --verify-only` — correctly fails with "did not report environment=staging" (response shows `environment=production`) - [x] **Positive test**: `./scripts/release/train-run.sh --verify-only` with default URL — staging gate passes (staging / -> 200, environment=staging confirmed) - [ ] End-to-end CI deployment: requires the WIF, IAM, and staging migration Job prerequisites above 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ls8g5a4bqcBgUzPMxSZ4ir [KAN-244]: https://tasteslikegood.atlassian.net/browse/KAN-244?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- Rovo Dev code review status --> --- Rovo Dev code review: <strong>Rovo Dev has reviewed this pull request</strong> Any suggestions or improvements have been posted as pull request comments. <!-- /Rovo Dev code review status -->
2 parents 2497c4d + 3b47182 commit 0bcc41b

3 files changed

Lines changed: 282 additions & 2 deletions

File tree

‎.github/workflows/staging-deploy.yml‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
name: Staging — build and deploy
2+
3+
# Rebuilds both images from the dev tip and deploys them to the staging
4+
# Cloud Run services whenever code merges to dev. This keeps staging
5+
# tracking the integration branch without manual intervention (KAN-244).
6+
7+
on:
8+
push:
9+
branches: [dev]
10+
paths-ignore:
11+
- 'specs/**'
12+
- 'docs/**'
13+
- '**.md'
14+
- '.claude/**'
15+
workflow_dispatch:
16+
17+
concurrency:
18+
group: staging-deploy
19+
# NOT cancel-in-progress. Flask and Express deploy in sequence, so cancelling
20+
# mid-run can leave staging with a new Flask image and the old Express image
21+
# (or a migrated DB and no deployed code) until the next run finishes — a
22+
# split-brain window of minutes. Queueing costs latency; cancelling costs
23+
# correctness.
24+
cancel-in-progress: false
25+
26+
permissions:
27+
contents: read
28+
id-token: write
29+
30+
env:
31+
STAGING_PROJECT: gen-lang-client-0491022701
32+
# Deliberately CROSS-PROJECT: images live in the production project's
33+
# Artifact Registry, while the services run in the staging project. The
34+
# staging Cloud Run service agent therefore needs artifactregistry.reader on
35+
# comdottasteslikegood — without it the deploy succeeds and the revision then
36+
# fails to pull, which surfaces as a healthy-looking deploy serving the old
37+
# image. The preflight step below turns that into an explicit failure.
38+
IMAGE_REGISTRY: us-central1-docker.pkg.dev/comdottasteslikegood/vegangenius
39+
REGION: us-central1
40+
41+
jobs:
42+
build-and-deploy:
43+
name: Build images + deploy to staging
44+
runs-on: ubuntu-latest
45+
steps:
46+
- uses: actions/checkout@v7
47+
with:
48+
submodules: recursive
49+
50+
- name: Authenticate to Google Cloud
51+
uses: google-github-actions/auth@v3
52+
with:
53+
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
54+
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
55+
56+
- name: Set up gcloud CLI
57+
uses: google-github-actions/setup-gcloud@v3
58+
59+
- name: Configure Docker for Artifact Registry
60+
run: gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
61+
62+
- name: Determine image tag
63+
id: tag
64+
run: echo "tag=staging-${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
65+
66+
# Cross-project pull preflight: fail here with a named cause rather than
67+
# letting the Cloud Run revision fail to pull later, which reads as a
68+
# green deploy still serving the previous image.
69+
- name: Preflight — staging can read the production Artifact Registry
70+
run: |
71+
if ! gcloud artifacts repositories describe vegangenius \
72+
--location="$REGION" --project=comdottasteslikegood >/dev/null 2>&1; then
73+
echo "::error::Cannot read Artifact Registry 'vegangenius' in" \
74+
"comdottasteslikegood. The staging Cloud Run service agent needs" \
75+
"roles/artifactregistry.reader on that project."
76+
exit 1
77+
fi
78+
79+
# ── Build ────────────────────────────────────────────────────────
80+
- name: Build Express frontend image
81+
run: docker build -t "$IMAGE_REGISTRY/express-frontend:${{ steps.tag.outputs.tag }}" .
82+
83+
- name: Build Flask backend image
84+
run: docker build -t "$IMAGE_REGISTRY/flask-backend:${{ steps.tag.outputs.tag }}" -f Backend/Dockerfile Backend
85+
86+
# ── Push ─────────────────────────────────────────────────────────
87+
- name: Push Express frontend image
88+
run: docker push "$IMAGE_REGISTRY/express-frontend:${{ steps.tag.outputs.tag }}"
89+
90+
- name: Push Flask backend image
91+
run: docker push "$IMAGE_REGISTRY/flask-backend:${{ steps.tag.outputs.tag }}"
92+
93+
# ── Migrate ──────────────────────────────────────────────────────
94+
# Must run BEFORE the Flask image rolls out, mirroring production, where
95+
# cloudbuild.yaml runs the flask-backend-migrate Job between image push
96+
# and service deploy. Without this a schema change deploys against a
97+
# stale staging DB and fails at runtime ("recipe.status missing" style),
98+
# which is exactly the class of failure staging exists to catch first.
99+
#
100+
# Fails closed. A staging deploy that cannot migrate must not ship a
101+
# Flask image against an unmigrated schema — the old revision keeps
102+
# serving instead, which is the desired failure.
103+
- name: Run Alembic migrations against staging
104+
run: |
105+
if ! gcloud run jobs describe flask-staging-migrate \
106+
--region="$REGION" --project="$STAGING_PROJECT" >/dev/null 2>&1; then
107+
echo "::error::Cloud Run Job 'flask-staging-migrate' does not exist in" \
108+
"$STAGING_PROJECT/$REGION. Staging cannot migrate, so this deploy is" \
109+
"refused rather than shipping Flask against a possibly stale schema." \
110+
"Create the Job (mirroring the production flask-backend-migrate Job)" \
111+
"and re-run."
112+
exit 1
113+
fi
114+
# `jobs execute` cannot change the image. Update the Job to the just-built
115+
# backend image and execute it in one blocking command; `--wait` implies
116+
# `--execute-now` for `jobs update`.
117+
gcloud run jobs update flask-staging-migrate \
118+
--region="$REGION" \
119+
--project="$STAGING_PROJECT" \
120+
--image="$IMAGE_REGISTRY/flask-backend:${{ steps.tag.outputs.tag }}" \
121+
--wait \
122+
--quiet
123+
124+
# ── Deploy (image swap only — no IAM or secret mutations) ───────
125+
- name: Deploy Flask backend to staging
126+
run: |
127+
gcloud run services update flask-backend-staging \
128+
--image="$IMAGE_REGISTRY/flask-backend:${{ steps.tag.outputs.tag }}" \
129+
--region="$REGION" \
130+
--project="$STAGING_PROJECT" \
131+
--quiet
132+
133+
- name: Deploy Express frontend to staging
134+
run: |
135+
gcloud run services update express-frontend-staging \
136+
--image="$IMAGE_REGISTRY/express-frontend:${{ steps.tag.outputs.tag }}" \
137+
--region="$REGION" \
138+
--project="$STAGING_PROJECT" \
139+
--quiet
140+
141+
# ── Verify ───────────────────────────────────────────────────────
142+
# No pre-emptive `sleep` here: the smoke test below already retries with
143+
# backoff, so a fixed sleep only adds latency to the common case.
144+
145+
- name: Resolve staging URL
146+
id: url
147+
run: |
148+
EXPRESS_URL=$(gcloud run services describe express-frontend-staging \
149+
--region="$REGION" --project="$STAGING_PROJECT" \
150+
--format='value(status.url)')
151+
echo "express=$EXPRESS_URL" >> "$GITHUB_OUTPUT"
152+
153+
- name: Smoke test staging
154+
run: |
155+
URL="${{ steps.url.outputs.express }}"
156+
echo "Staging URL: $URL"
157+
158+
# Retry for cold start. Only sleep BETWEEN attempts — sleeping after
159+
# the final failure just burns 10s before the job fails anyway.
160+
ATTEMPTS=3
161+
for attempt in $(seq 1 "$ATTEMPTS"); do
162+
CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$URL/" || echo "000")
163+
if [ "$CODE" = "200" ]; then break; fi
164+
if [ "$attempt" -lt "$ATTEMPTS" ]; then
165+
echo "Attempt $attempt: got $CODE, retrying in 10s..."
166+
sleep 10
167+
else
168+
echo "Attempt $attempt: got $CODE"
169+
fi
170+
done
171+
172+
if [ "$CODE" != "200" ]; then
173+
echo "::error::Staging / returned $CODE after $ATTEMPTS attempts"
174+
exit 1
175+
fi
176+
echo "Staging / -> 200"
177+
178+
# Verify environment=staging (guards against URL misconfiguration).
179+
# Parsed with jq for an exact field match: a substring grep can be
180+
# satisfied by the string appearing anywhere in the body.
181+
HEALTH=$(curl -sf --max-time 15 "$URL/api/health" || echo "")
182+
ENVIRONMENT=$(printf '%s' "$HEALTH" | jq -r '.environment // empty' 2>/dev/null || echo "")
183+
if [ "$ENVIRONMENT" = "staging" ]; then
184+
echo "Staging /api/health reports environment=staging"
185+
else
186+
echo "::error::Staging /api/health reported environment='$ENVIRONMENT', expected 'staging'"
187+
echo "Response: $HEALTH"
188+
exit 1
189+
fi
190+
191+
# /api/health is served by Express BEFORE the proxy (server/index.ts,
192+
# "Health check (local to Express — handled before the proxy)"), so it
193+
# returns 200 even when Flask is completely down. It proves Express is
194+
# up and correctly configured, and nothing more.
195+
#
196+
# /browse is server-rendered by Flask against the staging Cloud SQL
197+
# instance, so it is the probe that actually exercises Flask + the DB.
198+
# It is a hard failure, not a warning: a staging deploy whose backend
199+
# cannot serve a DB-backed page has not succeeded, and warning-only
200+
# meant a broken Flask backend still reported a green deploy.
201+
BROWSE_CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$URL/browse" || echo "000")
202+
if [ "$BROWSE_CODE" != "200" ]; then
203+
echo "::error::Staging /browse returned $BROWSE_CODE — Flask or the staging" \
204+
"database is unreachable. Express being up is not sufficient."
205+
exit 1
206+
fi
207+
echo "Staging /browse -> 200 (Flask + Cloud SQL reachable)"

‎scripts/release/RUNBOOK.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,12 @@ scoped to one).
193193
./scripts/release/train-run.sh --verify-only # or the manual form below
194194
```
195195

196+
`--verify-only` runs a **staging health gate first**: it checks that the staging
197+
Cloud Run pair returns 200 and that `/api/health` reports `environment=staging`
198+
(guarding against `STAGING_URL` accidentally pointing at production). The default
199+
URL is hardcoded to the known staging service; override with `STAGING_URL` if the
200+
service is redeployed to a new URL.
201+
196202
> **TRAP — verify the code, not the bundle name, and not `main-*.js` alone.**
197203
> On v0.4.8 the deploy was live while a poller grepping only `main-*.js` reported
198204
> "not deployed" for twenty minutes. The app is code-split: `publishFailureMessage`

‎scripts/release/train-run.sh‎

Lines changed: 69 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,13 @@ STATE="$STATE_DIR/train-state.json"
4242
BACKEND_REPO="adamtasteslikegood/tasteslikegood.com"
4343
PROD="https://www.tasteslikegood.org"
4444

45+
# Staging URL. Overridable via STAGING_URL env. Defaults to the known
46+
# Cloud Run URL so --verify-only works without gcloud credentials. If the
47+
# service is redeployed to a new URL, either export STAGING_URL or update
48+
# this constant — there is deliberately no gcloud fallback (would require
49+
# auth in every operator's shell).
50+
STAGING="${STAGING_URL:-https://express-frontend-staging-g24svmewaa-uc.a.run.app}"
51+
4552
DRY_RUN=0
4653
MODE="walk"
4754
MARKER=""
@@ -358,6 +365,50 @@ print_checklist() {
358365
done
359366
}
360367

368+
# ── staging health gate ────────────────────────────────────────────────────────
369+
# Blocks --verify-only until staging returns 200 and reports environment=staging.
370+
# Retries handle the min-instances=0 cold start. Runs before verify_prod() so a
371+
# staging outage surfaces before anyone looks at production.
372+
verify_staging() {
373+
head2 "Staging health gate"
374+
375+
info "staging: $STAGING"
376+
377+
local attempt code body
378+
local max_attempts=3
379+
380+
# Check / returns 200 (with cold-start retries).
381+
for attempt in $(seq 1 $max_attempts); do
382+
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$STAGING/" || echo "000")
383+
if [ "$code" = "200" ]; then break; fi
384+
if [ "$attempt" -lt "$max_attempts" ]; then
385+
info "staging / returned $code (attempt $attempt/$max_attempts, retrying in 5s)"
386+
sleep 5
387+
fi
388+
done
389+
if [ "$code" = "200" ]; then
390+
ok "staging / → 200"
391+
else
392+
bad "staging / → $code after $max_attempts attempts"
393+
return 1
394+
fi
395+
396+
# /api/health must report environment=staging. This guards against
397+
# STAGING_URL accidentally pointing at production, which would make the
398+
# whole gate silently meaningless.
399+
body=$(curl -sf --max-time 15 "$STAGING/api/health" 2>/dev/null || echo "")
400+
# Anchor with `[,}]` so a substring elsewhere in the JSON cannot satisfy the guard.
401+
if echo "$body" | grep -qE '"environment"[[:space:]]*:[[:space:]]*"staging"[[:space:]]*[,}]'; then
402+
ok "staging /api/health reports environment=staging"
403+
else
404+
bad "staging /api/health did not report environment=staging"
405+
info "response: $body"
406+
return 1
407+
fi
408+
409+
ok "staging health gate passed"
410+
}
411+
361412
# ── step 9: production verification ─────────────────────────────────────────
362413
# Greps EVERY served asset, not main-*.js. On v0.4.8 the deploy was live while a
363414
# main-only poller reported "not deployed" for twenty minutes, because the
@@ -530,8 +581,24 @@ gather
530581
derive_steps
531582

532583
if [ "$MODE" = "verify" ]; then
533-
verify_prod "$MARKER"
534-
exit $?
584+
# Staging is checked FIRST but never gates the production check. --verify-only
585+
# exists to answer "is this build actually live in production?", and a staging
586+
# outage is precisely when an operator most needs that answer. Refusing to look
587+
# at production because staging is down withholds the one fact being asked for.
588+
#
589+
# The gate keeps its teeth: a staging failure still makes the command exit
590+
# non-zero, it just does so after production has been verified and reported.
591+
staging_rc=0
592+
verify_staging || staging_rc=1
593+
[ "$staging_rc" -eq 0 ] || bad "staging health gate failed — continuing to the production check anyway"
594+
595+
prod_rc=0
596+
verify_prod "$MARKER" || prod_rc=1
597+
598+
if [ "$prod_rc" -ne 0 ] || [ "$staging_rc" -ne 0 ]; then
599+
exit 1
600+
fi
601+
exit 0
535602
fi
536603

537604
if [ "$MODE" = "bump" ]; then

0 commit comments

Comments
 (0)