Skip to content

Add CI performance budgets and canary metrics checks #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shayancoin wants to merge 3 commits into from
Closed

Add CI performance budgets and canary metrics checks #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9aee543
Add CI performance budgets and canary metrics checks
shayancoin Oct 16, 2025
9ba2fc4
Handle missing Prometheus data and guard previous error logging
shayancoin Oct 16, 2025
9c26b9b
📝 Add docstrings to `codex/extend-ci-pipeline-with-performance-thresh…
coderabbitai[bot] Oct 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 125 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,128 @@ jobs:

- name: Build
run: npm run build

perf-budgets:
runs-on: ubuntu-latest
needs:
- backend-tests
- frontend-tests
timeout-minutes: 40
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8

- name: Set up Node.js
uses: actions/setup-node@0a44ba78451273a1ed8ac2fee4e347c72dfd377f
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: ./frontend/package-lock.json

- name: Install dependencies
working-directory: ./frontend
run: npm ci

- name: Install Playwright browsers
working-directory: ./frontend
run: npx playwright install --with-deps chromium

- name: Start stack
run: docker compose --env-file .env.development -f docker-compose.dev.yml up -d --build
Comment on lines +115 to +131
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Install Python before using it in perf-budgets job

The job invokes python to seed the backend but never sets up Python. On hosted runners, python may not be available as python (often python3 only), causing a hard failure.

Apply this diff to add setup-python (mirrors earlier jobs):

       - name: Set up Node.js
         uses: actions/setup-node@0a44ba78451273a1ed8ac2fee4e347c72dfd377f
         with:
           node-version: '20'
           cache: 'npm'
           cache-dependency-path: ./frontend/package-lock.json
+
+      - name: Set up Python
+        uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c
+        with:
+          python-version: '3.12'
🤖 Prompt for AI Agents 
.github/workflows/ci.yml around lines 115 to 131: the perf-budgets job runs
Python to seed the backend but never installs or configures Python, which can
fail on hosted runners where only python3 is available; add a step before any
python invocation that uses actions/setup-python (e.g. actions/setup-python@v4)
with python-version: '3.x' (or a specific 3.11/3.10 as used elsewhere) so the
runner exposes the python binary, and mirror any cache/config from earlier jobs
if needed.


- name: Wait for API
run: |
for i in {1..60}; do curl -sf http://localhost:8000/healthcheck && break || sleep 2; done

- name: Wait for Frontend
run: |
for i in {1..60}; do curl -sf http://localhost:3000/models/manifest.json && break || sleep 2; done

Comment on lines +133 to +140
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Make readiness loops fail if services never become ready

Current for-loops can exit successfully after the final sleep even if curl never succeeds, letting the job proceed in a bad state.

Apply this diff to both API and Frontend waits:

-      - name: Wait for API
-        run: |
-          for i in {1..60}; do curl -sf http://localhost:8000/healthcheck && break || sleep 2; done
+      - name: Wait for API
+        run: |
+          for i in {1..60}; do
+            if curl -sf http://localhost:8000/healthcheck; then
+              echo "API ready"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "API failed to become ready in time" >&2
+          exit 1
@@
-      - name: Wait for Frontend
-        run: |
-          for i in {1..60}; do curl -sf http://localhost:3000/models/manifest.json && break || sleep 2; done
+      - name: Wait for Frontend
+        run: |
+          for i in {1..60}; do
+            if curl -sf http://localhost:3000/models/manifest.json; then
+              echo "Frontend ready"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "Frontend failed to become ready in time" >&2
+          exit 1
🤖 Prompt for AI Agents 
In .github/workflows/ci.yml around lines 133 to 140, the readiness for-loops
currently can complete quietly even if curl never succeeded; change each block
so the workflow fails when the service never becomes ready by adding an explicit
failure after the loop (e.g., run the curl check once more and if it still fails
echo an error and exit 1, or set a flag inside the loop and after the loop test
it and exit 1 on failure), applying this to both the API and Frontend wait
steps.

- name: Seed backend for perf
env:
BASE_URL: http://localhost:8000
run: |
python - <<'PY'
import json
import os
import urllib.error
import urllib.request

BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000")

def post(path: str, payload: dict) -> dict:
req = urllib.request.Request(
f"{BASE_URL}{path}",
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
detail = exc.read().decode("utf-8", "ignore")
raise SystemExit(f"Seed request failed ({exc.code}): {detail}")

material = post(
"/api/materials/",
{"name": "Walnut", "texture_url": None, "cost_per_sq_ft": 12.5},
)
material_id = material.get("id")
if not material_id:
raise SystemExit("Material creation failed; missing id")

post(
"/api/modules/",
{
"name": "Base600",
"width": 600.0,
"height": 720.0,
"depth": 580.0,
"base_price": 100.0,
"material_id": material_id,
},
)
PY
Comment on lines +145 to +185
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Fix YAML block: indent heredoc body consistently (current YAML parse error)

Lines inside the run: | block after “python - <<'PY'” are not indented, so YAML treats them as new keys. Indent the entire heredoc body consistently to satisfy YAML while preserving the script.

Apply this diff:

       - name: Seed backend for perf
         env:
           BASE_URL: http://localhost:8000
         run: |
-          python - <<'PY'
-import json
-import os
-import urllib.error
-import urllib.request
-
-BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000")
-
-def post(path: str, payload: dict) -> dict:
-    req = urllib.request.Request(
-        f"{BASE_URL}{path}",
-        data=json.dumps(payload).encode("utf-8"),
-        headers={"Content-Type": "application/json"},
-    )
-    try:
-        with urllib.request.urlopen(req, timeout=10) as resp:
-            return json.loads(resp.read().decode("utf-8"))
-    except urllib.error.HTTPError as exc:
-        detail = exc.read().decode("utf-8", "ignore")
-        raise SystemExit(f"Seed request failed ({exc.code}): {detail}")
-
-material = post(
-    "/api/materials/",
-    {"name": "Walnut", "texture_url": None, "cost_per_sq_ft": 12.5},
-)
-material_id = material.get("id")
-if not material_id:
-    raise SystemExit("Material creation failed; missing id")
-
-post(
-    "/api/modules/",
-    {
-        "name": "Base600",
-        "width": 600.0,
-        "height": 720.0,
-        "depth": 580.0,
-        "base_price": 100.0,
-        "material_id": material_id,
-    },
-)
-PY
+          python - <<'PY'
+          import json
+          import os
+          import urllib.error
+          import urllib.request
+
+          BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000")
+
+          def post(path: str, payload: dict) -> dict:
+              req = urllib.request.Request(
+                  f"{BASE_URL}{path}",
+                  data=json.dumps(payload).encode("utf-8"),
+                  headers={"Content-Type": "application/json"},
+              )
+              try:
+                  with urllib.request.urlopen(req, timeout=10) as resp:
+                      return json.loads(resp.read().decode("utf-8"))
+              except urllib.error.HTTPError as exc:
+                  detail = exc.read().decode("utf-8", "ignore")
+                  raise SystemExit(f"Seed request failed ({exc.code}): {detail}")
+
+          material = post(
+              "/api/materials/",
+              {"name": "Walnut", "texture_url": None, "cost_per_sq_ft": 12.5},
+          )
+          material_id = material.get("id")
+          if not material_id:
+              raise SystemExit("Material creation failed; missing id")
+
+          post(
+              "/api/modules/",
+              {
+                  "name": "Base600",
+                  "width": 600.0,
+                  "height": 720.0,
+                  "depth": 580.0,
+                  "base_price": 100.0,
+                  "material_id": material_id,
+              },
+          )
+          PY

Note: YAML will strip the common indent from the block scalar, so the shell sees the heredoc exactly as intended. The current error matches actionlint/YAMLlint output.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
python - <<'PY'
import json
import os
import urllib.error
import urllib.request
BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000")
def post(path: str, payload: dict) -> dict:
req = urllib.request.Request(
f"{BASE_URL}{path}",
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
detail = exc.read().decode("utf-8", "ignore")
raise SystemExit(f"Seed request failed ({exc.code}): {detail}")
material = post(
"/api/materials/",
{"name": "Walnut", "texture_url": None, "cost_per_sq_ft": 12.5},
)
material_id = material.get("id")
if not material_id:
raise SystemExit("Material creation failed; missing id")
post(
"/api/modules/",
{
"name": "Base600",
"width": 600.0,
"height": 720.0,
"depth": 580.0,
"base_price": 100.0,
"material_id": material_id,
},
)
PY
- name: Seed backend for perf
env:
BASE_URL: http://localhost:8000
run: |
python - <<'PY'
import json
import os
import urllib.error
import urllib.request
BASE_URL = os.environ.get("BASE_URL", "http://localhost:8000")
def post(path: str, payload: dict) -> dict:
req = urllib.request.Request(
f"{BASE_URL}{path}",
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
return json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
detail = exc.read().decode("utf-8", "ignore")
raise SystemExit(f"Seed request failed ({exc.code}): {detail}")
material = post(
"/api/materials/",
{"name": "Walnut", "texture_url": None, "cost_per_sq_ft": 12.5},
)
material_id = material.get("id")
if not material_id:
raise SystemExit("Material creation failed; missing id")
post(
"/api/modules/",
{
"name": "Base600",
"width": 600.0,
"height": 720.0,
"depth": 580.0,
"base_price": 100.0,
"material_id": material_id,
},
)
PY
🧰 Tools
🪛 actionlint (1.7.8)

146-146: could not parse as YAML: could not find expected ':'

(syntax-check)

🪛 YAMLlint (1.37.1)

[error] 147-147: syntax error: could not find expected ':'

(syntax)

🤖 Prompt for AI Agents 
In .github/workflows/ci.yml around lines 145 to 185 the python heredoc started
with "python - <<'PY'" is not indented, causing YAML to treat the script lines
as top-level keys; indent the entire heredoc body (every line between the
"python - <<'PY'" line and the closing "PY") to the same indentation level as
the run block content (e.g., two extra spaces) so YAML parses the block scalar
correctly, and keep the closing "PY" delimiter indented to that same level; do
not change the script content otherwise (YAML will strip common indentation when
passing the heredoc to the shell).


- name: Run performance budgets
run: npm run --prefix frontend perf:budget

- name: Upload perf budget results
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: perf-budget-results
path: |
perf-results/perf-budget-summary.json
perf-results/perf-budget-junit.xml

- name: Publish performance budget summary
if: always()
uses: actions/upload-test-results@0c62d1d6f6cfaf4c5859e1b358a5d2df4f96701a
with:
files: perf-results/perf-budget-junit.xml

- name: Shutdown stack
if: always()
run: docker compose --env-file .env.development -f docker-compose.dev.yml down

canary-metrics:
runs-on: ubuntu-latest
needs:
- perf-budgets
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8

- name: Evaluate canary metrics
env:
CANARY_METRICS_FIXTURE: tests/perf/canary-metrics.fixture.json
P95_THRESHOLD_MS: '3000'
ERROR_RATE_THRESHOLD: '0.02'
REGRESSION_TOLERANCE_PCT: '0.1'
run: python scripts/ci/check_canary_metrics.py

Comment on lines +209 to +223
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Install Python before running canary metrics script

This job runs python scripts/ci/check_canary_metrics.py without provisioning Python. Add setup-python or invoke python3 only if guaranteed to exist.

Apply this diff:

   canary-metrics:
     runs-on: ubuntu-latest
     needs:
       - perf-budgets
     steps:
       - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
+
+      - name: Set up Python
+        uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c
+        with:
+          python-version: '3.12'
@@
-      - name: Evaluate canary metrics
+      - name: Evaluate canary metrics
         env:
           CANARY_METRICS_FIXTURE: tests/perf/canary-metrics.fixture.json
           P95_THRESHOLD_MS: '3000'
           ERROR_RATE_THRESHOLD: '0.02'
           REGRESSION_TOLERANCE_PCT: '0.1'
         run: python scripts/ci/check_canary_metrics.py
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
canary-metrics:
runs-on: ubuntu-latest
needs:
- perf-budgets
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
- name: Evaluate canary metrics
env:
CANARY_METRICS_FIXTURE: tests/perf/canary-metrics.fixture.json
P95_THRESHOLD_MS: '3000'
ERROR_RATE_THRESHOLD: '0.02'
REGRESSION_TOLERANCE_PCT: '0.1'
run: python scripts/ci/check_canary_metrics.py
canary-metrics:
runs-on: ubuntu-latest
needs:
- perf-budgets
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
- name: Set up Python
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c
with:
python-version: '3.12'
- name: Evaluate canary metrics
env:
CANARY_METRICS_FIXTURE: tests/perf/canary-metrics.fixture.json
P95_THRESHOLD_MS: '3000'
ERROR_RATE_THRESHOLD: '0.02'
REGRESSION_TOLERANCE_PCT: '0.1'
run: python scripts/ci/check_canary_metrics.py
🤖 Prompt for AI Agents 
In .github/workflows/ci.yml around lines 209 to 223, the canary-metrics job
invokes python scripts/ci/check_canary_metrics.py without ensuring Python is
available; add a step before the run that sets up Python (e.g.
actions/setup-python@v4 with a python-version like '3.x' or a specific '3.11')
so the runner has a known python binary on PATH, then keep the existing run step
(or change the run to invoke python3 explicitly if you prefer not to add the
setup action).

- name: Upload canary metrics summary
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: canary-metrics-summary
path: perf-results/canary-metrics-summary.json
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ out/
.env
.env.*
!.env.example
perf-results/

# Docker
**/.dockerignore
Expand Down
66 changes: 66 additions & 0 deletions .perf-budget.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
version: 2
defaults:
run_count: 3
throttling:
profile: slow-4g
cpu_slowdown_multiplier: 4
download_throughput_kbps: 1500
upload_throughput_kbps: 750
request_latency_ms: 40
scenarios:
- id: configurator-load
description: Directly load the configurator with seeded data
url: http://localhost:3000/configurator
waits:
- type: selector
selector: "canvas"
timeout_ms: 60000
- type: networkidle
idle_ms: 5000
timeout_ms: 60000
selectors:
viewer_canvas: "canvas"
generate_button: "button:has-text(\"Generate Quote\")"
price_total: "text=Total:"
metrics:
- id: first-contentful-paint
aggregation: p75
threshold: 2000
unit: ms
- id: largest-contentful-paint
aggregation: p75
threshold: 3500
unit: ms
- id: total-blocking-time
aggregation: p75
threshold: 200
unit: ms
- id: cumulative-layout-shift
aggregation: p75
threshold: 0.1
unit: score
- id: homepage-to-configurator
description: Navigate from the marketing homepage into the configurator experience
run_count: 5
steps:
- type: goto
url: http://localhost:3000/
wait_until: networkidle
- type: wait_for_selector
selector: "main"
timeout_ms: 30000
- type: goto
url: http://localhost:3000/configurator
wait_until: networkidle
- type: wait_for_selector
selector: "canvas"
timeout_ms: 60000
metrics:
- id: navigation-duration
aggregation: p90
threshold: 3000
unit: ms
- id: largest-contentful-paint
aggregation: p95
threshold: 4000
unit: ms
54 changes: 54 additions & 0 deletions docs/release-checklist.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Release Checklist

This checklist connects the CI performance signals introduced in this repository with the
runtime observability tooling that guards production releases. It is intended for release
managers and on-call engineers who need a repeatable flow that ties Grafana alerts, Tempo
traces, and CI preview environments together.

## 1. Validate CI performance budgets

1. Confirm the **Performance Budgets** job succeeded in the latest CI run.
* Inspect the JUnit summary uploaded to the run (artifact `perf-budget-results`).
* Review `perf-results/perf-budget-summary.json` for the concrete values captured by
Playwright (P75 configurator load, P90 navigation duration, P95 LCP).
2. If any threshold failed, investigate before promoting the release candidate:
* Re-run the job against the preview environment to determine whether the regression
is deterministic or environment-specific.
* Capture a Grafana dashboard snapshot showing the relevant Web Vitals panel and link
it in the incident tracker.

## 2. Check canary latency and error regressions

1. Verify the **Canary Metrics** job completed without failures.
2. Review `perf-results/canary-metrics-summary.json` to compare the current build's P95
latency and error rate with the previous build and the configured budgets.
3. If the job failed:
* Acknowledge or silence the corresponding Grafana alert for the service-level
objective (SLO).
* Escalate to the on-call engineer through the paging integration configured for the
Grafana alert rule (OpsGenie, PagerDuty, etc.).
* Use the Tempo trace search link emitted by the job (or Grafana Explore) to confirm
whether elevated latency correlates with specific spans.

## 3. Correlate CI results with Grafana dashboards

1. Publish the artifacts from the CI run (`perf-budget-summary.json` and
`canary-metrics-summary.json`) to the shared release channel or ticket.
2. Update the Grafana dashboard annotations with the build ID and a link to the CI run so
that on-call engineers can quickly correlate spikes with the release candidate.
3. Ensure the Grafana dashboard panels for "CI Build Budgets" and "Runtime Latency" use the
Pushgateway/JUnit exports for side-by-side comparisons.

## 4. Preview gating and release sign-off

1. Block the promotion of the preview environment to staging/production until both
performance-related jobs have succeeded for the release branch commit.
2. Confirm no active Grafana alerts remain open for the release window. If alerts exist,
document mitigations and obtain sign-off from the incident commander before proceeding.
3. Log the final decision in the release record, linking to:
* The CI run with passing performance jobs.
* Grafana alert history or dashboards showing the green state.
* Tempo traces or logs that justify the decision when applicable.

Following this checklist guarantees that CI regressions, Grafana alerts, and on-call
notifications remain aligned, providing a defensible audit trail for each production push.
11 changes: 6 additions & 5 deletions frontend/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 6 additions & 7 deletions frontend/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
"start": "next start",
"lint": "next lint",
"test:e2e": "playwright test",
"perf:budget": "node --loader ts-node/esm tests/perf/run-perf-budget.ts",
"assets:gen": "python ../scripts/generate_reference_glbs.py",
"assets:pack": "bash ../scripts/pack_models.sh",
"assets:validate": "python ../scripts/glb_validate.py public/models/*.glb --fail-on-warning",
Expand All @@ -20,15 +21,15 @@
"@chakra-ui/react": "^2.8.2",
"@emotion/react": "^11.13.3",
"@emotion/styled": "^11.13.0",
"@react-three/drei": "^9.120.5",
"@react-three/fiber": "^8.17.10",
"framer-motion": "^11.5.4",
"next": "^14.2.28",
"react": "^18",
"react-dom": "^18",
"react-icons": "^5.3.0",
"zustand": "^4.5.0",
"three": "^0.171.0",
"@react-three/fiber": "^8.17.10",
"@react-three/drei": "^9.120.5"
"zustand": "^4.5.0"
},
"devDependencies": {
"@playwright/test": "^1.56.0",
Expand All @@ -49,12 +50,10 @@
"postcss": "^8",
"tailwindcss": "^3.3.0",
"ts-jest": "^29.2.5",
"ts-node": "^10.9.2",
"typescript": "^5",
"gltfpack": "0.25.0",
"meshoptimizer": "0.25.0",
"vitest": "^1.6.0",
"@playwright/test": "^1.56.0",
"ts-node": "^10.9.2"
"yaml": "^2.8.1"
},
"jest": {
"setupFilesAfterEnv": [
Expand Down
Loading