Skip to content

Commit fc037e1

Browse files
smocklesmockle
authored
feat: Add E2E tests (#25)
Part of github/continuous-ai-for-accessibility#18 (Hubber access only) Depends on github-community-projects/continuous-ai-for-accessibility-scanner#24 This PR adds a new “Test” workflow that runs an end-to-end test: 1. A Jekyll-powered site—with known issues—is served. 2. That site is scanned using 'github/continuous-ai-for-accessibility-scanner', and scan results are cached. 3. The cache is inspected to ensure all known issues were discovered. 4. Each issue and pull request opened by the scanner is checked too. 5. Finally, all issues and pull requests are closed, and the cache is wiped. I successfully tested this workflow in another repo: https://github.com/github/accessibility-sandbox/actions/runs/17145745125 (Hubber access only). It also ran successfully here: https://github.com/github/continuous-ai-for-accessibility-scanner/actions/runs/17146422836 (and check out the associated [issues](https://github.com/github/continuous-ai-for-accessibility-scanner/issues?q=is%3Aissue%20state%3Aclosed%20created%3A2025-08-22%20%22Accessibility%20issue%3A%22%20in%3Atitle) and [pull requests](https://github.com/github/continuous-ai-for-accessibility-scanner/pulls?q=is%3Apr+is%3Aclosed+created%3A2025-08-22+author%3Acopilot-swe-agent%5Bbot%5D+))! After this PR is merged, I’ll update our [branch policy](https://github.com/github/continuous-ai-for-accessibility-scanner/settings/branch_protection_rules/65848741) to require this “Test” workflow. After that, I’ll work on adding some of the additional test cases outlined in github/continuous-ai-for-accessibility#18 (Hubber access only).
2 parents 0389939 + 77df1d8 commit fc037e1

File tree

13 files changed

+2442
-0
lines changed

13 files changed

+2442
-0
lines changed

‎.github/workflows/test.yml‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
name: Test
2+
on:
3+
pull_request:
4+
types:
5+
- synchronize
6+
push:
7+
branches:
8+
- main
9+
workflow_dispatch:
10+
11+
permissions:
12+
contents: write
13+
issues: write
14+
pull-requests: write
15+
16+
jobs:
17+
test:
18+
name: Test
19+
runs-on: ubuntu-latest
20+
strategy:
21+
matrix:
22+
site: [ "sites/site-with-errors" ]
23+
steps:
24+
- name: Checkout
25+
uses: actions/checkout@v5
26+
27+
- name: Setup Ruby
28+
uses: ruby/setup-ruby@829114fc20da43a41d27359103ec7a63020954d4
29+
with:
30+
ruby-version: "3.4"
31+
bundler-cache: true
32+
working-directory: ${{ matrix.site }}
33+
34+
- name: Start Jekyll (${{ matrix.site }})
35+
shell: bash
36+
working-directory: ${{ matrix.site }}
37+
env:
38+
JEKYLL_ENV: production
39+
run: |
40+
set -euo pipefail
41+
bundle exec jekyll serve --no-watch --detach
42+
echo "Starting Jekyll on port 4000"
43+
curl -fsS --retry 25 --retry-delay 1 --retry-all-errors "http://127.0.0.1:4000/" > /dev/null
44+
echo "Jekyll has started"
45+
46+
- name: Generate cache key
47+
id: cache_key
48+
shell: bash
49+
run: |
50+
echo "cache_key=$(printf 'cached_findings-%s-%s.json' "${{ github.ref_name }}" "${{ matrix.site }}" | tr -cs 'A-Za-z0-9._-' '_')" >> $GITHUB_OUTPUT
51+
52+
- name: Scan site (${{ matrix.site }})
53+
# FIXME: Switch branch to 'main' after https://github.com/github/continuous-ai-for-accessibility-scanner/pull/24 is merged
54+
uses: github/continuous-ai-for-accessibility-scanner@smockle/cache_key
55+
with:
56+
urls: |
57+
http://127.0.0.1:4000/
58+
http://127.0.0.1:4000/jekyll/update/2025/07/30/welcome-to-jekyll.html
59+
http://127.0.0.1:4000/about/
60+
http://127.0.0.1:4000/404.html
61+
repository: github/continuous-ai-for-accessibility-scanner
62+
token: ${{ secrets.GH_TOKEN }}
63+
cache_key: ${{ steps.cache_key.outputs.cache_key }}
64+
65+
- name: Retrieve cached findings
66+
# FIXME: Switch branch to 'main' after https://github.com/github/continuous-ai-for-accessibility-scanner/pull/24 is merged
67+
uses: github/continuous-ai-for-accessibility-scanner/.github/actions/gh-cache/restore@smockle/cache_key
68+
with:
69+
path: ${{ steps.cache_key.outputs.cache_key }}
70+
token: ${{ secrets.GITHUB_TOKEN }}
71+
72+
- name: Add PR URLs to findings
73+
uses: actions/github-script@v7
74+
with:
75+
github-token: ${{ secrets.GITHUB_TOKEN }}
76+
script: |
77+
const fs = require('fs');
78+
if (!process.env.FINDINGS_PATH || !fs.existsSync(process.env.FINDINGS_PATH)) {
79+
core.info("Skipping 'Add PR URLs to findings' (no cached findings).");
80+
return;
81+
}
82+
const findings = JSON.parse(fs.readFileSync(process.env.FINDINGS_PATH, 'utf-8'));
83+
for (const finding of findings) {
84+
if (!finding.issueUrl) {
85+
continue;
86+
}
87+
const { owner, repo, issueNumber } = /https:\/\/github\.com\/(?<owner>[^/]+)\/(?<repo>[^/]+)\/issues\/(?<issueNumber>\d+)/.exec(finding.issueUrl).groups;
88+
const query = `query($owner: String!, $repo: String!, $issueNumber: Int!) {
89+
repository(owner: $owner, name: $repo) {
90+
issue(number: $issueNumber) {
91+
timelineItems(first: 100, itemTypes: [CONNECTED_EVENT, CROSS_REFERENCED_EVENT]) {
92+
nodes {
93+
... on CrossReferencedEvent { source { ... on PullRequest { url } } }
94+
... on ConnectedEvent { subject { ... on PullRequest { url } } }
95+
}
96+
}
97+
}
98+
}
99+
}`;
100+
const variables = { owner, repo, issueNumber: parseInt(issueNumber, 10) }
101+
const result = await github.graphql(query, variables)
102+
let pullRequest;
103+
if ('source' in result.repository?.issue?.timelineItems.nodes[0]) {
104+
pullRequest = result.repository?.issue?.timelineItems.nodes[0].source;
105+
} else if ('subject' in result.repository?.issue?.timelineItems.nodes[0]) {
106+
pullRequest = result.repository?.issue?.timelineItems.nodes[0].subject;
107+
}
108+
if (pullRequest) {
109+
finding.pullRequestUrl = pullRequest.url;
110+
}
111+
}
112+
fs.writeFileSync(process.env.FINDINGS_PATH, JSON.stringify(findings));
113+
env:
114+
FINDINGS_PATH: ${{ steps.cache_key.outputs.cache_key }}
115+
116+
- name: Validate scan results (${{ matrix.site }})
117+
run: |
118+
npm ci
119+
npm run test
120+
env:
121+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
122+
FINDINGS_PATH: ${{ steps.cache_key.outputs.cache_key }}
123+
124+
- name: Clean up issues and pull requests
125+
if: ${{ always() }}
126+
shell: bash
127+
run: |
128+
set -euo pipefail
129+
if [[ ! -f "${{ steps.cache_key.outputs.cache_key }}" ]]; then
130+
echo "Skipping 'Clean up issues and pull requests' (no cached findings)."
131+
exit 0
132+
fi
133+
jq -r '
134+
(if type=="string" then fromjson else . end)
135+
| .[] | .issueUrl, .pullRequestUrl
136+
| select(. != null)
137+
' "${{ steps.cache_key.outputs.cache_key }}" \
138+
| while read -r URL; do
139+
if [[ "$URL" == *"/pull/"* ]]; then
140+
echo "Closing pull request: $URL"
141+
gh pr close "$URL" || echo "Failed to close pull request: $URL"
142+
elif [[ "$URL" == *"/issues/"* ]]; then
143+
echo "Closing issue: $URL"
144+
gh issue close "$URL" || echo "Failed to close issue: $URL"
145+
else
146+
echo "Skipping unrecognized url: $URL"
147+
fi
148+
done
149+
env:
150+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
151+
152+
- name: Clean up cached findings
153+
if: ${{ always() }}
154+
# FIXME: Switch branch to 'main' after https://github.com/github/continuous-ai-for-accessibility-scanner/pull/24 is merged
155+
uses: github/continuous-ai-for-accessibility-scanner/.github/actions/gh-cache/delete@smockle/cache_key
156+
with:
157+
path: ${{ steps.cache_key.outputs.cache_key }}
158+
token: ${{ secrets.GITHUB_TOKEN }}

0 commit comments

Comments
 (0)