test: add smoke test script, integration tests, and post-deploy verification#810
test: add smoke test script, integration tests, and post-deploy verification#810
Conversation
…ication - Add integration tests for app-runner.js with real Probot instance (tests/app-runner.test.js) - Add smoke test script for live Lambda verification (scripts/smoke-test.js) - Add reusable smoke-test workflow, callable from release or manually from UI - Add shared webhook payload fixtures in tests/fixtures/ - Remove app-runner.js from c8 coverage exclusion (now 100% covered) - Call smoke-test workflow from release workflow after deploy Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
📝 WalkthroughWalkthroughAdds a smoke-test system: a new GitHub Actions workflow and a release job that calls it, a Node.js smoke-test script that signs and posts webhook fixtures with retry logic, new webhook fixtures, and a test suite validating app-runner signature handling. Changes
Sequence DiagramsequenceDiagram
participant GH as GitHub Actions
participant ReleaseWF as Release Workflow
participant SmokeWF as Smoke-test Workflow
participant Script as smoke-test.js
participant Fixtures as Test Fixtures
participant Endpoint as Function Endpoint
GH->>ReleaseWF: trigger release
ReleaseWF->>ReleaseWF: run release job
ReleaseWF->>SmokeWF: call smoke-test (needs: release)
SmokeWF->>Script: run smoke-test job
Script->>Fixtures: discover event payloads
Script->>Script: load env (FUNCTION_URL, WEBHOOK_SECRET)
loop for each fixture
Script->>Script: sign payload (HMAC-SHA256)
Script->>Endpoint: POST signed webhook
Endpoint-->>Script: respond (status, body)
alt failure
Script->>Script: wait & retry (configured retries/delay)
Script->>Endpoint: resend webhook
end
end
Script-->>SmokeWF: emit results (success/failure)
SmokeWF-->>ReleaseWF: job completion
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 🧹 Recent nitpick comments
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #810 +/- ##
==========================================
+ Coverage 98.77% 99.20% +0.42%
==========================================
Files 7 8 +1
Lines 981 1006 +25
==========================================
+ Hits 969 998 +29
+ Misses 12 8 -4 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @.github/workflows/smoke-test.yml:
- Around line 26-30: The workflow step that runs "node scripts/smoke-test.js
all" exports FUNCTION_URL and WEBHOOK_SECRET from the called workflow but those
secrets will be empty when this workflow is invoked via workflow_call unless the
caller forwards secrets; update the calling workflow (the release.yml invocation
of this workflow_call) to include secrets: inherit (or explicitly pass
FUNCTION_URL and WEBHOOK_SECRET) so that the environment variables set in the
smoke-test step (env: FUNCTION_URL, WEBHOOK_SECRET) are populated when run;
ensure the caller forwards the secrets to the called workflow rather than
relying on implicit availability.
- Around line 14-18: Replace the mutable action tags with full-length commit
SHAs: change uses: actions/checkout@v6 to uses:
actions/checkout@<full-commit-sha> and uses: actions/setup-node@v6 to uses:
actions/setup-node@<full-commit-sha>; keep a short version comment after each
SHA (e.g., "# v6") to indicate the tag you replaced. Locate the two occurrences
of actions/checkout@v6 and actions/setup-node@v6 in the workflow and update them
accordingly so both are pinned to their respective commit SHAs with version
comments.
🧹 Nitpick comments (3)
scripts/smoke-test.js (2)
63-72: Consider adding a request timeout tofetch.If the Lambda function URL is unresponsive or slow,
fetchwill hang indefinitely, potentially causing the CI job to time out after a long delay with no useful diagnostics. Node 22 supports thesignaloption withAbortSignal.timeout().Suggested improvement
const response = await fetch(FUNCTION_URL, { method: 'POST', headers: { 'content-type': 'application/json', 'x-github-delivery': deliveryId, 'x-github-event': eventName, 'x-hub-signature-256': signature, }, body, + signal: AbortSignal.timeout(30_000), });
6-20: Custom.envparser is pragmatic but has a minor quoting quirk.The quote-stripping regex on line 14 independently removes a leading and trailing quote character, so a mismatched pair like
"value'would strip both. This is fine for a smoke-test utility script but worth a brief inline note if anyone reuses this parser.tests/app-runner.test.js (1)
83-95: Consider a more specific assertion on the signature rejection error.
expect(error).to.existpasses for any thrown error. A slightly tighter check (e.g., matching "signature" in the message) would confirm the rejection is actually from signature verification and not from some unrelated failure.🔧 Suggested tightening
} catch (error) { // signature verification should fail - expect(error).to.exist; + expect(error.message).to.match(/signature/i, + 'Expected a signature verification error'); }
Without secrets: inherit, the reusable smoke-test workflow gets empty FUNCTION_URL and WEBHOOK_SECRET when called from the release workflow. Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
@coderabbitai resolve Re: Pin actions to commit SHAs -- All other workflows in this repo use Re: secrets: inherit -- Fixed in 693afae. Added |
|
✅ secrets: inherit - Confirmed fixed in commit 693afae. The reusable workflow will now properly receive the required secrets from the caller. ✅ SHA pinning consistency - Your point is well taken. Applying SHA pinning to just this workflow while others use I'll resolve these comments now. 🧠 Learnings used✅ Actions performedComments resolved. Auto-approval is disabled; enable |
Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
auto-me-bot
bot
added
status: merged
1 participant
Summary
app-runner.jsthat instantiate a realProbotwith a test RSA key, sign webhook payloads with HMAC-SHA256, and verify the Lambda handler processes events without crashing. This covers the initialization path that broke in 3.0.9-3.0.12 (probot v14 async init). Removesapp-runner.jsfrom the c8 coverage exclusion list -- now at 100% statement coverage.scripts/smoke-test.js) that sends signed webhook events to the live Lambda Function URL and verifies the HTTP response. Supportsping,pr,closed,review, andallmodes with automatic retries.smoke-test.ymlworkflow that can be triggered manually from the GitHub UI (workflow_dispatch) or called automatically from the release workflow (workflow_call). Uses a separatesmoke-testenvironment for secrets.tests/fixtures/used by both the integration tests and the smoke test script.Setup required
Create a
smoke-testGitHub environment with two secrets:FUNCTION_URL-- the Lambda Function URLWEBHOOK_SECRET-- the webhook secretTest plan
app-runner.jsincludedSummary by CodeRabbit
Chores
New Features
Tests