Skip to content

Commit 2cee3df

Browse files
TomerFicursoragent
andauthored
test: add smoke test script, integration tests, and post-deploy verification (#810)
* test: add smoke test script, integration tests, and post-deploy verification - Add integration tests for app-runner.js with real Probot instance (tests/app-runner.test.js) - Add smoke test script for live Lambda verification (scripts/smoke-test.js) - Add reusable smoke-test workflow, callable from release or manually from UI - Add shared webhook payload fixtures in tests/fixtures/ - Remove app-runner.js from c8 coverage exclusion (now 100% covered) - Call smoke-test workflow from release workflow after deploy Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: add secrets: inherit so smoke-test workflow receives secrets Without secrets: inherit, the reusable smoke-test workflow gets empty FUNCTION_URL and WEBHOOK_SECRET when called from the release workflow. Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: retrigger PR checks Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Tomer Figenblat <tomer@figenblat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 9dcfb85 commit 2cee3df

File tree

10 files changed

+394
-3
lines changed

10 files changed

+394
-3
lines changed

.github/workflows/release.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,3 +148,8 @@ jobs:
148148
git add mkdocs.yml
149149
git commit -m "docs: updated docs with ${{ steps.bumper.outputs.next }} [skip ci]"
150150
git push
151+
152+
smoke-test:
153+
needs: release
154+
uses: ./.github/workflows/smoke-test.yml
155+
secrets: inherit

.github/workflows/smoke-test.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
name: Smoke Test
3+
4+
on:
5+
workflow_dispatch:
6+
workflow_call:
7+
8+
jobs:
9+
smoke-test:
10+
runs-on: ubuntu-latest
11+
environment: smoke-test
12+
name: Smoke test deployed function
13+
steps:
14+
- name: Checkout sources
15+
uses: actions/checkout@v6
16+
17+
- name: Install node 22
18+
uses: actions/setup-node@v6
19+
with:
20+
node-version: '22'
21+
22+
- name: Wait for propagation
23+
if: github.event_name != 'workflow_dispatch'
24+
run: sleep 10
25+
26+
- name: Run smoke tests
27+
env:
28+
FUNCTION_URL: ${{ secrets.FUNCTION_URL }}
29+
WEBHOOK_SECRET: ${{ secrets.WEBHOOK_SECRET }}
30+
run: node scripts/smoke-test.js all

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ lambda.zip
1010
auto-me-bot.zip
1111
unit-tests-result.json
1212
release.sh
13+
plans/

package.json

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,9 +62,7 @@
6262
"include": [
6363
"src/**/*.js"
6464
],
65-
"exclude": [
66-
"src/app-runner.js"
67-
],
65+
"exclude": [],
6866
"reporter": [
6967
"html",
7068
"lcov",

scripts/smoke-test.js

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
import { createHmac, randomUUID } from 'node:crypto';
2+
import { readFileSync, readdirSync } from 'node:fs';
3+
import { resolve, dirname, basename } from 'node:path';
4+
import { fileURLToPath } from 'node:url';
5+
6+
// load .env if present (no external dependencies, optional for CI)
7+
const __dirname = dirname(fileURLToPath(import.meta.url));
8+
const envPath = resolve(__dirname, '..', '.env');
9+
try {
10+
for (const line of readFileSync(envPath, 'utf-8').split('\n')) {
11+
const match = line.match(/^([^#=]+)=(.*)$/);
12+
if (match) {
13+
const key = match[1].trim();
14+
const val = match[2].trim().replace(/^["']|["']$/g, '');
15+
if (!process.env[key]) process.env[key] = val;
16+
}
17+
}
18+
} catch {
19+
// no .env file -- rely on environment variables (e.g. in CI)
20+
}
21+
22+
const FUNCTION_URL = process.env.FUNCTION_URL;
23+
const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET;
24+
25+
if (!FUNCTION_URL) {
26+
console.error('FUNCTION_URL is not set in .env');
27+
process.exit(1);
28+
}
29+
if (!WEBHOOK_SECRET) {
30+
console.error('WEBHOOK_SECRET is not set in .env');
31+
process.exit(1);
32+
}
33+
34+
// discover available events from payloads directory
35+
// filename format: <event-name>.json where dots in event name map to dots in filename
36+
// e.g. pull_request.opened.json -> event "pull_request", action "opened"
37+
// ping.json -> event "ping", no action
38+
const payloadsDir = resolve(__dirname, '..', 'tests', 'fixtures');
39+
const EVENTS = {};
40+
for (const file of readdirSync(payloadsDir).filter(f => f.endsWith('.json'))) {
41+
const label = basename(file, '.json');
42+
const parts = label.split('.');
43+
// first part before the dot is the github event name (e.g. "pull_request", "pull_request_review")
44+
// but event names can contain underscores, so we use the payload's action field to determine the split
45+
const payload = JSON.parse(readFileSync(resolve(payloadsDir, file), 'utf-8'));
46+
const eventName = payload.action ? label.slice(0, label.lastIndexOf('.')) : label;
47+
EVENTS[label] = { name: eventName, payload };
48+
}
49+
50+
// sign a payload the same way GitHub does
51+
function sign(payload) {
52+
return 'sha256=' + createHmac('sha256', WEBHOOK_SECRET).update(payload).digest('hex');
53+
}
54+
55+
// send a webhook event to the function
56+
async function sendEvent(eventName, payload) {
57+
const body = JSON.stringify(payload);
58+
const deliveryId = randomUUID();
59+
const signature = sign(body);
60+
61+
console.log(`Sending '${eventName}' event (delivery: ${deliveryId})...`);
62+
63+
const response = await fetch(FUNCTION_URL, {
64+
method: 'POST',
65+
headers: {
66+
'content-type': 'application/json',
67+
'x-github-delivery': deliveryId,
68+
'x-github-event': eventName,
69+
'x-hub-signature-256': signature,
70+
},
71+
body,
72+
});
73+
74+
const responseBody = await response.text();
75+
return { status: response.status, body: responseBody };
76+
}
77+
78+
const MAX_RETRIES = Number(process.env.SMOKE_RETRIES) || 3;
79+
const RETRY_DELAY_MS = Number(process.env.SMOKE_RETRY_DELAY_MS) || 5000;
80+
81+
function sleep(ms) {
82+
return new Promise(resolve => setTimeout(resolve, ms));
83+
}
84+
85+
async function runOne(label, eventName, payload) {
86+
for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
87+
try {
88+
const { status, body } = await sendEvent(eventName, payload);
89+
if (status >= 200 && status < 300) {
90+
console.log(` ${label}: OK (${status})`);
91+
return true;
92+
}
93+
console.error(` ${label}: FAILED (${status})${attempt < MAX_RETRIES ? ' - retrying...' : ''}`);
94+
if (body) console.error(` Response: ${body}`);
95+
} catch (error) {
96+
console.error(` ${label}: ERROR - ${error.message}${attempt < MAX_RETRIES ? ' - retrying...' : ''}`);
97+
}
98+
if (attempt < MAX_RETRIES) await sleep(RETRY_DELAY_MS);
99+
}
100+
return false;
101+
}
102+
103+
async function main() {
104+
const arg = process.argv[2] || 'ping';
105+
const labels = Object.keys(EVENTS).sort();
106+
107+
if (arg !== 'all' && !(arg in EVENTS)) {
108+
console.error(`Unknown event type: ${arg}`);
109+
console.error(`Supported events: ${labels.join(', ')}, all`);
110+
process.exit(1);
111+
}
112+
113+
let failed = false;
114+
115+
if (arg === 'all') {
116+
console.log('Running all smoke tests...\n');
117+
for (const label of labels) {
118+
const event = EVENTS[label];
119+
const ok = await runOne(label, event.name, event.payload);
120+
if (!ok) failed = true;
121+
}
122+
console.log(failed ? '\nSome tests failed.' : '\nAll tests passed.');
123+
} else {
124+
const event = EVENTS[arg];
125+
const ok = await runOne(arg, event.name, event.payload);
126+
if (!ok) failed = true;
127+
}
128+
129+
process.exit(failed ? 1 : 0);
130+
}
131+
132+
main();

tests/app-runner.test.js

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
import { expect } from 'chai'
2+
import { createHmac, generateKeyPairSync, randomUUID } from 'node:crypto'
3+
import { readFileSync } from 'node:fs'
4+
import { resolve, dirname } from 'node:path'
5+
import { fileURLToPath } from 'node:url'
6+
7+
const __dirname = dirname(fileURLToPath(import.meta.url));
8+
const fixturesDir = resolve(__dirname, 'fixtures');
9+
10+
// generate a test RSA key pair (probot requires a valid private key)
11+
const { privateKey: testPrivateKey } = generateKeyPairSync('rsa', {
12+
modulusLength: 2048,
13+
privateKeyEncoding: { type: 'pkcs1', format: 'pem' },
14+
publicKeyEncoding: { type: 'pkcs1', format: 'pem' },
15+
});
16+
17+
const TEST_APP_ID = '12345';
18+
const TEST_WEBHOOK_SECRET = 'test-webhook-secret';
19+
const TEST_PRIVATE_KEY_B64 = Buffer.from(testPrivateKey).toString('base64');
20+
21+
// build a lambda event object from a fixture file
22+
function buildEvent(fixtureFile, eventName) {
23+
const payload = readFileSync(resolve(fixturesDir, fixtureFile), 'utf-8');
24+
const parsed = JSON.parse(payload);
25+
// derive event name from filename if not provided
26+
if (!eventName) {
27+
const base = fixtureFile.replace('.json', '');
28+
eventName = parsed.action ? base.slice(0, base.lastIndexOf('.')) : base;
29+
}
30+
const body = JSON.stringify(parsed);
31+
const signature = 'sha256=' + createHmac('sha256', TEST_WEBHOOK_SECRET)
32+
.update(body).digest('hex');
33+
34+
return {
35+
headers: {
36+
'x-github-delivery': randomUUID(),
37+
'x-github-event': eventName,
38+
'x-hub-signature-256': signature,
39+
},
40+
body,
41+
};
42+
}
43+
44+
suite('Testing the app-runner handler', () => {
45+
let originalEnv;
46+
47+
suiteSetup(() => {
48+
// save and override env vars needed by the handler
49+
originalEnv = { ...process.env };
50+
process.env.APP_ID = TEST_APP_ID;
51+
process.env.WEBHOOK_SECRET = TEST_WEBHOOK_SECRET;
52+
process.env.PRIVATE_KEY = TEST_PRIVATE_KEY_B64;
53+
process.env.LOG_LEVEL = 'fatal'; // suppress logs during tests
54+
});
55+
56+
suiteTeardown(() => {
57+
// restore original env
58+
process.env = originalEnv;
59+
});
60+
61+
test('Handler initializes probot and processes a ping event without throwing', async () => {
62+
const { handler } = await import('../src/app-runner.js');
63+
const event = buildEvent('ping.json', 'ping');
64+
// ping events have no matching handler, so this should resolve cleanly
65+
await handler(event);
66+
});
67+
68+
test('Handler initializes probot and accepts a pull_request event', async () => {
69+
const { handler } = await import('../src/app-runner.js');
70+
const event = buildEvent('pull_request.opened.json');
71+
// handlers will attempt GitHub API calls which will fail, but
72+
// verifyAndReceive should not reject -- errors are handled by onError
73+
try {
74+
await handler(event);
75+
} catch (error) {
76+
// even if handlers fail on API calls, the function should not crash
77+
// on initialization -- that's the critical path we're testing
78+
expect(error.message).to.not.match(/cannot read properties of null/i,
79+
'Probot initialization failed -- probot.log or probot.webhooks is null');
80+
}
81+
});
82+
83+
test('Handler rejects with an invalid webhook signature', async () => {
84+
const { handler } = await import('../src/app-runner.js');
85+
const event = buildEvent('ping.json', 'ping');
86+
// tamper with the signature
87+
event.headers['x-hub-signature-256'] = 'sha256=invalid';
88+
try {
89+
await handler(event);
90+
expect.fail('Expected handler to reject with invalid signature');
91+
} catch (error) {
92+
// signature verification should fail
93+
expect(error).to.exist;
94+
}
95+
});
96+
});

tests/fixtures/ping.json

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"zen": "Speak like a human.",
3+
"hook_id": 1,
4+
"hook": {
5+
"type": "App",
6+
"id": 1,
7+
"active": true,
8+
"events": ["pull_request"],
9+
"app_id": 1
10+
}
11+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{
2+
"action": "closed",
3+
"number": 1,
4+
"pull_request": {
5+
"number": 1,
6+
"title": "smoke test PR",
7+
"body": "- [x] smoke test task",
8+
"state": "closed",
9+
"draft": false,
10+
"merged": true,
11+
"head": {
12+
"sha": "0000000000000000000000000000000000000000",
13+
"ref": "smoke-test"
14+
},
15+
"base": {
16+
"ref": "main"
17+
},
18+
"user": {
19+
"login": "smoke-test",
20+
"type": "User"
21+
}
22+
},
23+
"repository": {
24+
"name": "auto-me-bot",
25+
"full_name": "TomerFi/auto-me-bot",
26+
"owner": {
27+
"login": "TomerFi"
28+
}
29+
},
30+
"sender": {
31+
"login": "smoke-test",
32+
"type": "User"
33+
},
34+
"installation": {
35+
"id": 1
36+
}
37+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{
2+
"action": "opened",
3+
"number": 1,
4+
"pull_request": {
5+
"number": 1,
6+
"title": "smoke test PR",
7+
"body": "- [ ] smoke test task",
8+
"state": "open",
9+
"draft": false,
10+
"merged": false,
11+
"head": {
12+
"sha": "0000000000000000000000000000000000000000",
13+
"ref": "smoke-test"
14+
},
15+
"base": {
16+
"ref": "main"
17+
},
18+
"user": {
19+
"login": "smoke-test",
20+
"type": "User"
21+
}
22+
},
23+
"repository": {
24+
"name": "auto-me-bot",
25+
"full_name": "TomerFi/auto-me-bot",
26+
"owner": {
27+
"login": "TomerFi"
28+
}
29+
},
30+
"sender": {
31+
"login": "smoke-test",
32+
"type": "User"
33+
},
34+
"installation": {
35+
"id": 1
36+
}
37+
}

0 commit comments

Comments
 (0)