Skip to content

Commit ae1b33b

Browse files
feat: add go flaky tests github action (#1013)
* feat: add Loki querying and flaky test analysis Add core functionality for analyzing test failures from Loki logs: - Query Loki using LogQL for test failure data - Parse and aggregate test failures across branches - Classify tests as flaky based on failure patterns - Generate detailed analysis reports with failure counts and workflow URLs - Configurable time ranges and result limits This initial implementation includes stub interfaces for Git and GitHub functionality that will be added in subsequent PRs. * refactor: simplify PR1 to focus only on Loki analysis Remove stub implementations and complexity that will be added in later PRs: - Remove GitClient and GitHubClient interfaces and stubs - Remove FilePath and RecentCommits from FlakyTest struct - Remove repository-directory and skip-posting-issues inputs - Remove author tracking and issue management code paths - Focus purely on Loki querying, log parsing, and flaky test detection This creates a clean foundation for PR2 (Git authors) and PR3 (GitHub issues) to build upon without unnecessary complexity in the initial implementation. * Add basic test coverage for Loki analysis functionality - Add MockLokiClient and MockFileSystem for testing - Test core AnalyzeFailures() functionality with valid Loki response - Test error handling when Loki client fails - Test ActionReport() method for both empty and populated reports - Test utility functions like generateSummary() and FlakyTest.String() - Use proper Loki response format with stream metadata for test data * Add comprehensive documentation and local development script - Added README.md with detailed usage instructions and how-it-works - Added CHANGELOG.md documenting features and implementation details - Added run-local.sh script for local development and testing - Documentation now matches original PR functionality * Remove test files * Format Go code with gofmt * Remove PR3-related features from documentation - Remove GitHub issue creation and management features - Remove dry run mode references - Remove GitHub CLI and issue template content - Documentation now covers Loki analysis and Git author tracking * Remove PR2-related features from documentation - Remove Git history analysis and author tracking features - Remove repository-directory input reference - Remove run-local.sh script usage, use go run directly - Documentation now covers only basic Loki analysis functionality * Update flaky test detection to support both main and master branches - Update aggregate.go to consider both 'main' and 'master' branches as indicators of flaky tests - Update README documentation to reflect main/master branch logic - Add comprehensive tests for master branch detection - Remove 'progressive PR structure' and 'technical details' from changelog * Update to new loki structure * Add .gitignore * Use tests from before * Remove github client * Rename action from analyze-test-failures to go-flaky-tests - Update action name and description - Rename directories and update all file references - Update module name and build paths - Update documentation and examples * Remove repository-directory parameter from local script The repository-directory parameter is not needed in the core action and was causing documentation inconsistency. * Remove github-token parameter from local script The github-token parameter is not used in the current implementation and was causing documentation inconsistency. * Fix linting issues - Fix prettier formatting for action.yaml, README.md, CHANGELOG.md - Add missing newlines at end of files - Remove test output file that shouldn't be committed * Remove _actual.json files * Run prettier again (?) * Remove test files * Include job name and attempt * Format test data * Clarify what scopes we need
1 parent efb9b27 commit ae1b33b

File tree

19 files changed

+2057
-0
lines changed

19 files changed

+2057
-0
lines changed

actions/go-flaky-tests/.gitignore

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Local development files
2+
.env
3+
test-failure-analysis.json
4+
5+
# Go build artifacts
6+
*.exe
7+
*.exe~
8+
*.dll
9+
*.so
10+
*.dylib
11+
12+
# Go test files
13+
*.test
14+
*_actual.json
15+
16+
# Go coverage files
17+
*.out
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Changelog
2+
3+
## [Unreleased]
4+
5+
### Added
6+
7+
- Initial implementation of flaky test analysis action
8+
- Loki integration for fetching test failure logs
9+
- Comprehensive test suite with golden file testing
10+
11+
### Features
12+
13+
- **Loki Log Analysis**: Fetches and parses test failure logs using LogQL
14+
- **Flaky Test Detection**: Identifies tests that fail inconsistently across branches
15+
- **Configurable Limits**: Top-K filtering to focus on most problematic tests

actions/go-flaky-tests/README.md

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Go Flaky Tests
2+
3+
A GitHub Action that detects and analyzes flaky Go tests by fetching logs from Loki.
4+
5+
## Features
6+
7+
- **Loki Integration**: Fetches test failure logs from Loki using LogQL queries
8+
- **Flaky Test Detection**: Identifies tests that fail inconsistently across different branches
9+
10+
## Usage
11+
12+
```yaml
13+
name: Go Flaky Tests
14+
on:
15+
schedule:
16+
- cron: "0 9 * * 1" # Run every Monday at 9 AM
17+
workflow_dispatch:
18+
19+
jobs:
20+
analyze-failures:
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- name: Go Flaky Tests
26+
uses: grafana/shared-workflows/actions/go-flaky-tests@main
27+
with:
28+
loki-url: ${{ secrets.LOKI_URL }}
29+
loki-username: ${{ secrets.LOKI_USERNAME }}
30+
loki-password: ${{ secrets.LOKI_PASSWORD }}
31+
repository: ${{ github.repository }}
32+
time-range: "7d"
33+
top-k: "5"
34+
```
35+
36+
## Inputs
37+
38+
| Input | Description | Required | Default |
39+
| --------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------- | ------- |
40+
| `loki-url` | Loki endpoint URL | ✅ | - |
41+
| `loki-username` | Username for Loki authentication | ❌ | - |
42+
| `loki-password` | Password for Loki authentication. If using Grafana Cloud, then the access policy for this token needs the `logs:read` scope. | ❌ | - |
43+
| `repository` | Repository name in 'owner/repo' format | ✅ | - |
44+
| `time-range` | Time range for the query (e.g., '1h', '24h', '7d') | ❌ | `1h` |
45+
| `top-k` | Include only the top K flaky tests by distinct branches count | ❌ | `3` |
46+
47+
## Outputs
48+
49+
| Output | Description |
50+
| ------------------ | ----------------------------------------------- |
51+
| `test-count` | Number of flaky tests found |
52+
| `analysis-summary` | Summary of the analysis results |
53+
| `report-path` | Path to the generated analysis report JSON file |
54+
55+
## How It Works
56+
57+
1. **Fetch Logs**: Queries Loki for test failure logs within the specified time range
58+
2. **Parse Failures**: Extracts test names, branches, and workflow URLs from logs
59+
3. **Detect Flaky Tests**: Identifies tests that fail on multiple branches or multiple times on main/master
60+
61+
## Flaky Test Detection Logic
62+
63+
A test is considered "flaky" if:
64+
65+
- It fails on the main or master branch, OR
66+
- It fails on multiple different branches
67+
68+
Tests that only fail on feature branches are not considered flaky, as they likely indicate legitimate test failures for that specific feature.
69+
70+
## Local Development
71+
72+
Run the analysis locally using the provided script:
73+
74+
```bash
75+
# Set required environment variables
76+
export LOKI_URL="your-loki-url"
77+
export REPOSITORY="owner/repo"
78+
export TIME_RANGE="24h"
79+
# Run the analysis
80+
go run ./cmd/go-flaky-tests
81+
```
82+
83+
## Requirements
84+
85+
- Go 1.22 or later
86+
- Access to Loki instance with test failure logs
87+
88+
## Output Format
89+
90+
The action generates a JSON report with the following structure:
91+
92+
```json
93+
{
94+
"test_count": 2,
95+
"analysis_summary": "Found 2 flaky tests. Most common tests: TestUserLogin (3 total failures; recently changed by alice), TestPayment (1 total failures; recently changed by bob)",
96+
"report_path": "/path/to/test-failure-analysis.json",
97+
"flaky_tests": [
98+
{
99+
"test_name": "TestUserLogin",
100+
"file_path": "handlers/auth_test.go",
101+
"total_failures": 3,
102+
"branch_counts": {
103+
"main": 2,
104+
"feature-branch": 1
105+
},
106+
"example_workflows": [
107+
"https://github.com/owner/repo/actions/runs/123",
108+
"https://github.com/owner/repo/actions/runs/124"
109+
],
110+
"recent_commits": [
111+
{
112+
"hash": "abc123",
113+
"author": "alice",
114+
"timestamp": "2024-01-15T10:30:00Z",
115+
"title": "Fix authentication flow"
116+
}
117+
]
118+
}
119+
]
120+
}
121+
```

actions/go-flaky-tests/action.yaml

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: "Go Flaky Tests"
2+
description: "Detect and analyze flaky Go tests using Loki logs"
3+
author: "Grafana Labs"
4+
5+
inputs:
6+
loki-url:
7+
description: "Loki endpoint URL"
8+
required: true
9+
loki-username:
10+
description: "Username for Loki authentication"
11+
required: false
12+
loki-password:
13+
description: "Password for Loki authentication"
14+
required: false
15+
repository:
16+
description: "Repository name in 'owner/repo' format (e.g., 'grafana/grafana')"
17+
required: true
18+
time-range:
19+
description: "Time range for the query (e.g., '1h', '24h', '7d')"
20+
required: false
21+
default: "1h"
22+
top-k:
23+
description: "Include only the top K flaky tests by distinct branches count in analysis"
24+
required: false
25+
default: "3"
26+
27+
runs:
28+
using: "composite"
29+
steps:
30+
- name: Set up Go
31+
uses: actions/setup-go@v5
32+
with:
33+
go-version: "1.22"
34+
35+
- name: Build and run analyzer
36+
shell: bash
37+
run: |
38+
cd ${{ github.action_path }}
39+
go build -o analyzer ./cmd/go-flaky-tests
40+
./analyzer
41+
env:
42+
LOKI_URL: ${{ inputs.loki-url }}
43+
LOKI_USERNAME: ${{ inputs.loki-username }}
44+
LOKI_PASSWORD: ${{ inputs.loki-password }}
45+
REPOSITORY: ${{ inputs.repository }}
46+
TIME_RANGE: ${{ inputs.time-range }}
47+
TOP_K: ${{ inputs.top-k }}
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"log"
6+
"slices"
7+
"strconv"
8+
"strings"
9+
)
10+
11+
type RawLogEntry struct {
12+
TestName string `json:"test_name"`
13+
Branch string `json:"branch"`
14+
WorkflowRunURL string `json:"workflow_run_url"`
15+
WorkflowJobName string `json:"workflow_job_name"`
16+
WorkflowRunAttempt int `json:"workflow_run_attempt"`
17+
}
18+
19+
func AggregateFlakyTestsFromResponse(lokiResp *LokiResponse) ([]FlakyTest, error) {
20+
var rawEntries []RawLogEntry
21+
for _, result := range lokiResp.Data.Result {
22+
testName := result.Stream["parent_test_name"]
23+
branch := result.Stream["ci_github_workflow_run_head_branch"]
24+
workflowRunURL := result.Stream["ci_github_workflow_run_html_url"]
25+
workflowJobName := result.Stream["ci_github_workflow_job_name"]
26+
workflowRunAttempt, _ := strconv.Atoi(result.Stream["ci_github_workflow_run_run_attempt"])
27+
28+
if testName == "" || branch == "" {
29+
continue
30+
}
31+
entry := RawLogEntry{
32+
TestName: testName,
33+
Branch: branch,
34+
WorkflowRunURL: workflowRunURL,
35+
WorkflowJobName: workflowJobName,
36+
WorkflowRunAttempt: workflowRunAttempt,
37+
}
38+
rawEntries = append(rawEntries, entry)
39+
}
40+
41+
log.Printf("🔄 Processed %d log lines, extracted %d valid test failure entries", len(lokiResp.Data.Result), len(rawEntries))
42+
43+
return detectFlakyTestsFromRawEntries(rawEntries), nil
44+
}
45+
46+
func detectFlakyTestsFromRawEntries(rawEntries []RawLogEntry) []FlakyTest {
47+
testMap := make(map[string]map[string]int)
48+
exampleWorkflows := make(map[string]map[GithubActionsWorkflow]bool)
49+
50+
for _, entry := range rawEntries {
51+
if entry.TestName == "" || entry.Branch == "" {
52+
continue
53+
}
54+
55+
if testMap[entry.TestName] == nil {
56+
testMap[entry.TestName] = make(map[string]int)
57+
exampleWorkflows[entry.TestName] = make(map[GithubActionsWorkflow]bool)
58+
}
59+
60+
testMap[entry.TestName][entry.Branch]++
61+
62+
workflow := GithubActionsWorkflow{
63+
RunURL: entry.WorkflowRunURL,
64+
JobName: entry.WorkflowJobName,
65+
Attempt: entry.WorkflowRunAttempt,
66+
}
67+
68+
if workflow != (GithubActionsWorkflow{}) && len(exampleWorkflows[entry.TestName]) < 3 {
69+
exampleWorkflows[entry.TestName][workflow] = true
70+
}
71+
}
72+
73+
var flakyTests []FlakyTest
74+
75+
for testName, branches := range testMap {
76+
isFlaky := false
77+
totalFailures := 0
78+
79+
for branch, count := range branches {
80+
totalFailures += count
81+
82+
if branch == "main" || branch == "master" {
83+
isFlaky = true
84+
}
85+
}
86+
87+
if len(branches) > 1 {
88+
isFlaky = true
89+
}
90+
91+
if !isFlaky {
92+
continue
93+
}
94+
95+
var branchSummary []string
96+
for branch, count := range branches {
97+
branchSummary = append(branchSummary, fmt.Sprintf("%s:%d", branch, count))
98+
}
99+
100+
var workflows []GithubActionsWorkflow
101+
for workflowURL := range exampleWorkflows[testName] {
102+
workflows = append(workflows, workflowURL)
103+
}
104+
105+
flakyTests = append(flakyTests, FlakyTest{
106+
TestName: testName,
107+
TotalFailures: totalFailures,
108+
BranchCounts: branches,
109+
ExampleWorkflows: workflows,
110+
})
111+
112+
log.Printf("🔍 Detected flaky test: %s (%d total failures) - branches: %s",
113+
testName, totalFailures, strings.Join(branchSummary, ", "))
114+
}
115+
116+
log.Printf("📈 Test analysis stats:")
117+
log.Printf(" - Total unique tests with failures: %d", len(testMap))
118+
log.Printf(" - Tests classified as flaky: %d", len(flakyTests))
119+
120+
return sortFlakyTests(flakyTests)
121+
}
122+
123+
func sortFlakyTests(tests []FlakyTest) []FlakyTest {
124+
slices.SortFunc(tests, func(a, b FlakyTest) int {
125+
branchesDelta := len(b.BranchCounts) - len(a.BranchCounts)
126+
if branchesDelta != 0 {
127+
return branchesDelta
128+
}
129+
if a.TestName < b.TestName {
130+
return -1
131+
} else if a.TestName > b.TestName {
132+
return 1
133+
}
134+
return 0
135+
})
136+
return tests
137+
}

0 commit comments

Comments
 (0)