github-actions: report flaky tests as GH issues#49778
github-actions: report flaky tests as GH issues#49778v1v wants to merge 1 commit intoelastic:mainfrom
Conversation
v1v
added
backport-skip
botelastic
bot
added
the
needs_team
|
This pull request doesn't have a |
🤖 GitHub commentsJust comment with:
|
📝 WalkthroughWalkthroughIntroduces a new GitHub Actions workflow file ( ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.github/workflows/buildkite-test-flaky.yml (1)
26-26: Consider ifmax-issues: '5'is sufficient.With daily execution and a 1-day analysis window, limiting to 5 concurrent issues may be restrictive if multiple tests are flaky. Consider whether this threshold aligns with expected flaky test frequency.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/buildkite-test-flaky.yml at line 26, The current GitHub Action configuration sets max-issues: '5' which may be too low for daily runs with a 1-day analysis window; update the max-issues setting (the max-issues: '5' entry) to a higher or configurable value that matches expected flaky test volume (e.g., increase the numeric limit, make it an environment/variable, or remove the hard cap) so the workflow can surface all relevant flaky-test issues instead of being limited to five.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/buildkite-test-flaky.yml:
- Around line 7-8: Workflow lacks the required GitHub permission for issue
management which causes authorization failures when the flaky-report action
creates/updates issues; update the permissions mapping to include issues: write
alongside contents: read so the action can create/update issues. Edit the
permissions block (the top-level permissions key) and add an entry "issues:
write" (or change permissions to "contents: read" and "issues: write") so the
flaky-report action has write access; ensure no other permission scopes are
unnecessarily broadened and keep the action name "flaky-report" in mind when
verifying the fix.
---
Nitpick comments:
In @.github/workflows/buildkite-test-flaky.yml:
- Line 26: The current GitHub Action configuration sets max-issues: '5' which
may be too low for daily runs with a 1-day analysis window; update the
max-issues setting (the max-issues: '5' entry) to a higher or configurable value
that matches expected flaky test volume (e.g., increase the numeric limit, make
it an environment/variable, or remove the hard cap) so the workflow can surface
all relevant flaky-test issues instead of being limited to five.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 62ad4081-b814-4075-b052-319079aabc0c
📒 Files selected for processing (1)
.github/workflows/buildkite-test-flaky.yml
| permissions: | ||
| contents: read |
There was a problem hiding this comment.
Missing issues: write permission.
The workflow creates/updates GitHub issues but only has contents: read permission. This will cause authorization failures when the flaky-report action attempts to create or update issues.
🔐 Proposed fix
permissions:
contents: read
+ issues: write📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| permissions: | |
| contents: read | |
| permissions: | |
| contents: read | |
| issues: write |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/buildkite-test-flaky.yml around lines 7 - 8, Workflow
lacks the required GitHub permission for issue management which causes
authorization failures when the flaky-report action creates/updates issues;
update the permissions mapping to include issues: write alongside contents: read
so the action can create/update issues. Edit the permissions block (the
top-level permissions key) and add an entry "issues: write" (or change
permissions to "contents: read" and "issues: write") so the flaky-report action
has write access; ensure no other permission scopes are unnecessarily broadened
and keep the action name "flaky-report" in mind when verifying the fix.
There was a problem hiding this comment.
That's incorrect, we use elastic/oblt-actions/github/create-token@v1 to get the ephemeral github tokens, that's used as an output
There was a problem hiding this comment.
@v1v, you're right — my mistake. The elastic/oblt-actions/github/create-token@v1 action generates an ephemeral GitHub App token with its own scopes/permissions, and that token's output is what's passed to flaky-report for issue management. The workflow-level permissions block only governs the default GITHUB_TOKEN, which isn't used for the issue operations here. No change needed.
✏️ Learnings added
Learnt from: v1v
Repo: elastic/beats PR: 49778
File: .github/workflows/buildkite-test-flaky.yml:7-8
Timestamp: 2026-03-30T17:42:57.640Z
Learning: In elastic/beats (and related elastic repos), workflows that use `elastic/oblt-actions/github/create-tokenv1` to generate an ephemeral GitHub App token do NOT need `issues: write` (or other elevated permissions) in the workflow-level `permissions` block. The ephemeral token carries its own permissions and is passed as a step output to downstream actions (e.g., `elastic/oblt-actions/buildkite/flaky-report`) for operations like creating/updating GitHub issues. The workflow-level `permissions` block only governs the default `GITHUB_TOKEN`.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: andrewkroh
Repo: elastic/beats PR: 0
File: :0-0
Timestamp: 2026-03-19T14:51:00.057Z
Learning: In the elastic/beats repository, always check that a PR includes a changelog fragment file in the `changelog/fragments/` directory. This applies to any PR that introduces features, bug fixes, or enhancements. The fragment is a YAML file created via `elastic-agent-changelog-tool new "Brief description"` and must contain:
- `kind`: type of change (e.g., `bug-fix`, `enhancement`, `feature`)
- `summary`: short user-facing description
- `component`: affected component (e.g., `filebeat`, `metricbeat`, `libbeat`)
- `issue`: (optional) URL to the related GitHub issue
The tool is installed with: `go install github.com/elastic/elastic-agent-changelog-toollatest`
Reference: docs/extend/index.md line 22.
| github-label: "flaky-test" | ||
| github-repo: ${{ github.repository }} | ||
| github-token: ${{ steps.fetch-token.outputs.token }} | ||
| max-issues: '5' |
There was a problem hiding this comment.
report up to 5 unique Test Flakies that happened in the last day.
1 participant
Proposed commit message
Automate the BK Test Result reporting to GH issues.
Then agentic workflows can triage and plan accordingly
Checklist
stresstest.shscript to run them under stress conditions and race detector to verify their stability../changelog/fragmentsusing the changelog tool.Disruptive User Impact
Author's Checklist
How to test this PR locally
Related issues
Requires elastic/oblt-actions#536
Use cases
Screenshots
Logs