Skip to content

Commit 4f14168

Browse files
google-labs-jules[bot]google-labs-jules[bot]
authored
Persist flaky monitor reports across PRs (#1417)
* feat: persist flaky test reports across PRs using workflow_run Refactors the flaky test monitoring to run in a separate workflow triggered by `workflow_run`. This allows the `github-test-reporter` to run in the context of the default branch, enabling it to access and maintain a history of test reports across different Pull Requests for accurate flaky test detection and historical comparison. - Removes `flaky-monitor` job from `tests.yml` - Adds `.github/workflows/flaky-monitor.yml` triggered on `workflow_run` of `Tests` - Configuring artifact download from the triggering run - Sets appropriate permissions for PR commenting and artifact access * feat: persist flaky test reports to orphan branch Implements a robust flaky test monitoring solution that persists test reports to an orphan branch (`flaky-monitor-results`) instead of relying on ephemeral workflow artifacts. - Creates `.github/workflows/flaky-monitor.yml` triggered by `workflow_run` on `Tests`. - Manages an orphan branch `flaky-monitor-results` to store historical test reports. - Configures the monitor to aggregate historical reports from the branch with the current run's artifacts. - Ensures only runs against the `main` branch update the persistent history. - Removes the ephemeral `flaky-monitor` job from `.github/workflows/tests.yml`. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
1 parent 9aa1a14 commit 4f14168

File tree

2 files changed

+129
-37
lines changed

2 files changed

+129
-37
lines changed

‎.github/workflows/flaky-monitor.yml‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
name: Flaky Monitor
2+
3+
on:
4+
workflow_run:
5+
workflows: [Tests]
6+
types:
7+
- completed
8+
9+
permissions:
10+
actions: read
11+
contents: write
12+
pull-requests: write
13+
checks: write
14+
15+
jobs:
16+
flaky-monitor:
17+
runs-on: ubuntu-latest
18+
if: ${{ github.event.workflow_run.conclusion == 'success' || github.event.workflow_run.conclusion == 'failure' }}
19+
steps:
20+
- name: 📥 Download CTRF Results
21+
uses: actions/download-artifact@v4
22+
with:
23+
run-id: ${{ github.event.workflow_run.id }}
24+
github-token: ${{ secrets.GITHUB_TOKEN }}
25+
pattern: ctrf-*
26+
path: ctrf-blob
27+
merge-multiple: true
28+
29+
- name: 📂 Checkout Flaky History
30+
uses: actions/checkout@v4
31+
continue-on-error: true
32+
with:
33+
ref: flaky-monitor-results
34+
path: flaky-history
35+
fetch-depth: 0
36+
37+
- name: 📦 Prepare Reports
38+
run: |
39+
mkdir -p all-reports
40+
41+
# initialize history branch if checkout failed (branch doesn't exist)
42+
if [ ! -d "flaky-history" ]; then
43+
echo "Creating orphan branch directory"
44+
mkdir -p flaky-history
45+
cd flaky-history
46+
git init
47+
git checkout --orphan flaky-monitor-results
48+
cd ..
49+
fi
50+
51+
# Copy history to processing dir
52+
# (Ignore errors if empty)
53+
cp flaky-history/*.json all-reports/ 2>/dev/null || true
54+
55+
# Process current run artifacts
56+
# Rename to avoid collisions and track run ID
57+
# We expect files like ctrf-tavern-ubuntu-latest.json
58+
for file in ctrf-blob/*.json; do
59+
if [ -e "$file" ]; then
60+
filename=$(basename "$file" .json)
61+
cp "$file" "all-reports/${filename}-${{ github.event.workflow_run.id }}.json"
62+
fi
63+
done
64+
65+
- name: Get PR Number
66+
id: pr_info
67+
run: |
68+
# Retrieve PR number for commenting
69+
# Note: workflow_run context has limited PR info depending on trigger
70+
echo "pr_number=${{ github.event.workflow_run.pull_requests[0].number }}" >> $GITHUB_OUTPUT
71+
72+
- name: 📊 Publish Test Report
73+
uses: ctrf-io/github-test-reporter@v1
74+
env:
75+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
76+
with:
77+
report-path: 'all-reports/*.json'
78+
upload-artifact: true
79+
fetch-previous-results: false # Using orphan branch instead
80+
pull-request: false
81+
issue: ${{ steps.pr_info.outputs.pr_number }}
82+
overwrite-comment: true
83+
84+
# Report Config
85+
summary-delta-report: true
86+
tests-changed-report: true
87+
insights-report: true
88+
slowest-report: true
89+
failed-folded-report: true
90+
flaky-rate-report: true
91+
fail-rate-report: true
92+
previous-results-report: true
93+
previous-results-max: 100
94+
artifact-name: 'flaky-report'
95+
96+
- name: 💾 Persist Results (Main Only)
97+
if: ${{ github.event.workflow_run.head_branch == 'main' }}
98+
run: |
99+
cd flaky-history
100+
101+
# Configure git identity
102+
git config user.name "github-actions[bot]"
103+
git config user.email "github-actions[bot]@users.noreply.github.com"
104+
105+
# Ensure remote is set up correctly regardless of checkout success/fail
106+
# First, remove origin if it exists to avoid errors on re-add
107+
git remote remove origin 2>/dev/null || true
108+
git remote add origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git
109+
110+
# Fetch origin to make sure we have the latest state if checkout failed earlier but branch exists
111+
# (This handles race conditions or edge cases where checkout failed for other reasons)
112+
git fetch origin flaky-monitor-results 2>/dev/null || true
113+
114+
# If we are on a detached head or master (from init), ensure we are on the orphan branch
115+
current_branch=$(git branch --show-current)
116+
if [ "$current_branch" != "flaky-monitor-results" ]; then
117+
git checkout -b flaky-monitor-results || git checkout flaky-monitor-results
118+
fi
119+
120+
# Copy NEW reports from current run to history
121+
# Note: We copy from the renamed versions in all-reports to ensure uniqueness
122+
# Filter for files ending in this run-id
123+
cp ../all-reports/*-${{ github.event.workflow_run.id }}.json .
124+
125+
git add .
126+
git commit -m "Add test results from run ${{ github.event.workflow_run.id }}"
127+
128+
# Push to the orphan branch
129+
git push origin flaky-monitor-results

‎.github/workflows/tests.yml‎

Lines changed: 0 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -188,40 +188,3 @@ jobs:
188188
with:
189189
name: ctrf-ui-${{ matrix.os }}
190190
path: tavern/internal/www/ctrf-ui-${{ matrix.os }}.json
191-
192-
flaky-monitor:
193-
needs: [tavern, implants, ui-tests]
194-
runs-on: ubuntu-latest
195-
if: always()
196-
steps:
197-
- name: 📥 Download CTRF Results
198-
uses: actions/download-artifact@v4
199-
with:
200-
pattern: ctrf-*
201-
path: ctrf-reports
202-
merge-multiple: true
203-
204-
- name: 📊 Publish Test Report
205-
uses: ctrf-io/github-test-reporter@v1
206-
if: always()
207-
env:
208-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
209-
with:
210-
report-path: 'ctrf-reports/*.json'
211-
upload-artifact: true
212-
fetch-previous-results: true
213-
pull-request: true
214-
overwrite-comment: true
215-
216-
# Report Config
217-
# https://github.com/ctrf-io/github-test-reporter/blob/main/docs/report-showcase.md
218-
summary-delta-report: true
219-
tests-changed-report: true
220-
insights-report: true
221-
slowest-report: true
222-
failed-folded-report: true
223-
flaky-rate-report: true
224-
fail-rate-report: true
225-
previous-results-report: true
226-
previous-results-max: 100
227-
artifact-name: 'flaky-report'

0 commit comments

Comments
 (0)