Skip to content

Commit 2b72e34

Browse files
clean up docs (#249)
1 parent d9dea62 commit 2b72e34

File tree

7 files changed

+129
-38
lines changed

7 files changed

+129
-38
lines changed

README.md

Lines changed: 127 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -19,45 +19,43 @@ You can support the project by:
1919
- Giving this repository a ⭐
2020
- [Following the @ctrf organization on GitHub](https://github.com/ctrf-io)
2121

22-
## Example
23-
24-
<div align="center">
25-
<img src="images/github-readme.png" alt="GitHub Test Reporter" width="600">
26-
</div>
27-
2822
## Key Features
2923

3024
**🧩 Post anywhere:** job summaries, pull requests, checks, issues, inline annotations, and other developer tools
3125

32-
**📊 Built-in insights:** failures, flaky tests, and trends across hundreds of runs
26+
**📊 Built-in insights:** failures, flaky tests, and duration trends across hundreds of runs
3327

3428
**🧘 Super flexible:** start fast with powerful built-in reports or go fully custom with your own templates
3529

36-
**🤖 Continuous AI:** summarize failed tests and get explanations of them with contextual insights
30+
**🤖 Continuous AI:** summarize test results with AI and get explanations of them with contextual insights
3731

3832
**🔌 Framework-agnostic:** works with any testing tool
3933

4034
## Table of Contents
4135

42-
1. [Usage](#usage)
43-
2. [Report Showcase](#report-showcase)
44-
3. [Visual Overview](#visual-overview)
45-
4. [Available Inputs](#available-inputs)
46-
5. [Generating an AI Report](#generating-an-ai-report)
47-
6. [Pull Requests](#pull-requests)
48-
7. [Status Checks](#status-checks)
49-
8. [Build Your Own Report](#build-your-own-report)
50-
9. [Customizing Report Order](#customizing-report-order)
51-
10. [Community Reports](#community-reports)
52-
11. [GitHub Token](#github-token)
53-
12. [Storing Artifacts](#storing-artifacts)
54-
13. [Filtering](#filtering)
55-
14. [Integrations](#integrations)
56-
15. [What is CTRF?](#what-is-ctrf)
57-
58-
## Usage
59-
60-
To get started add the following to your workflow file:
36+
1. [Basic Usage](#basic-usage)
37+
2. [Insights Usage](#insights-usage)
38+
3. [Continuous AI Usage](#continuous-ai-usage)
39+
4. [Pull Request Comment Usage](#pull-request-comment-usage)
40+
5. [Super Flexible Usage](#super-flexible-usage)
41+
6. [Report Showcase](#report-showcase)
42+
7. [Generate a report](#generate-a-report)
43+
8. [Available Inputs](#available-inputs)
44+
9. [Generating an AI Report](#generating-an-ai-report)
45+
10. [Pull Requests](#pull-requests)
46+
11. [Status Checks](#status-checks)
47+
12. [Build Your Own Report](#build-your-own-report)
48+
13. [Customizing Report Order](#customizing-report-order)
49+
14. [Community Reports](#community-reports)
50+
15. [GitHub Token](#github-token)
51+
16. [Storing Artifacts](#storing-artifacts)
52+
17. [Filtering](#filtering)
53+
18. [Integrations](#integrations)
54+
19. [What is CTRF?](#what-is-ctrf)
55+
56+
## Basic Usage
57+
58+
To get started quickly add the following to your workflow file:
6159

6260
```yaml
6361
- name: Publish Test Report
@@ -68,15 +66,106 @@ To get started add the following to your workflow file:
6866
if: always()
6967
```
7068
71-
## Report Showcase
69+
Add `upload-artifact: true`, `fetch-previous-results: true` and a `GITHUB_TOKEN` to enable comparisons.
70+
71+
<div align="center">
72+
<img src="images/github.png" alt="GitHub Test Reporter" width="600">
73+
</div>
74+
75+
## Insights Usage
76+
77+
To get started with multi run insights on failures, flakiness and test duration, add the following to your workflow file:
78+
79+
```yaml
80+
- name: Publish Test Report with Insights
81+
uses: ctrf-io/github-test-reporter@v1
82+
with:
83+
report-path: './ctrf/*.json'
84+
summary-delta-report: true
85+
insights-report: true
86+
flaky-rate-report: true
87+
fail-rate-report: true
88+
slowest-report: true
89+
upload-artifact: true
90+
env:
91+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
92+
if: always()
93+
```
94+
95+
This will create a job summary with insights on failures, flakiness and test duration.
96+
97+
<div align="center">
98+
<img src="images/insights.png" alt="GitHub Test Reporter" width="600">
99+
</div>
100+
101+
## Continuous AI Usage
102+
103+
To get started with continuous AI analysis of test results, add the following to your workflow file:
104+
105+
```yaml
106+
- name: Publish Test Report with AI Analysis
107+
uses: ctrf-io/github-test-reporter@v1
108+
with:
109+
report-path: './ctrf/*.json'
110+
ai-summary-report: true
111+
on-fail-only: true
112+
summary: false
113+
pull-request: true
114+
ai: |
115+
{
116+
"provider": "openai",
117+
"model": "gpt-5"
118+
}
119+
if: always()
120+
env:
121+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
122+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
123+
```
124+
125+
Add your AI API key to the repository secrets. Choose from a variety of AI providers and models. See [AI Configuration](#ai-configuration) for more information.
126+
127+
Make sure your GitHub Token has pull request write permission.
72128

73-
Checkout the [built-in reports](docs/report-showcase.md)
129+
This will create a pull request comment with a summary of the AI analysis.
74130

75-
## Visual Overview
131+
<div align="center">
132+
<img src="images/ai-summary.png" alt="GitHub Test Reporter" width="600">
133+
</div>
76134

77-
| ![all](images/all.png) | ![github](images/github-failed.png) | ![flaky-rate](images/insights.png) | ![historical](images/historical.png) | ![pr](images/pr.png) |
78-
| :--------------------------------------: | :----------------------------------------: | :----------------------------------: | :--------------------------: | :----------------------------------: |
79-
| ![suite-folded](images/suite-folded.png) | ![ai](images/ai.png) | ![custom](images/custom.png) | ![failed](images/failed.png) | ![suite-list](images/suite-list.png) |
135+
## Pull Request Comment Usage
136+
137+
To get started with pull request comments, add the following to your workflow file:
138+
139+
```yaml
140+
- name: Publish Test Report with Pull Request Comment
141+
uses: ctrf-io/github-test-reporter@v1
142+
with:
143+
report-path: './ctrf/*.json'
144+
pull-request-report: true
145+
if: always()
146+
env:
147+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
148+
```
149+
150+
Make sure your GitHub Token has pull request write permission.
151+
152+
This will create a pull request comment with a summary of the test results.
153+
154+
<div align="center">
155+
<img src="images/pr-comment.png" alt="GitHub Test Reporter" width="600">
156+
</div>
157+
158+
<div align="center">
159+
<img src="images/pr-comment-with-fail.png" alt="GitHub Test Reporter" width="600">
160+
</div>
161+
162+
## Super Flexible Usage
163+
164+
These are just a few examples of the many ways the GitHub Test Reporter can be used. It is super flexible and can be used in a variety of ways. Continue reading to learn more about the many features and options available.
165+
166+
## Report Showcase
167+
168+
There are a variety of built-in reports to choose from. Checkout the [built-in reports](docs/report-showcase.md)
80169

81170
## Generate a report
82171

@@ -86,6 +175,8 @@ This reporter works best with CTRF as it's packed with modern properties. [CTRF
86175

87176
For more details on using JUnit, see [JUnit integration instructions](docs/integrations.md#junit-to-ctrf-integration)
88177

178+
Don't see a reporter for your testing framework? Consider contributing 💚
179+
89180
## Available Inputs
90181

91182
For more advanced usage, there are several inputs available.
@@ -207,6 +298,7 @@ The following AI providers are supported:
207298
- **perplexity** - Perplexity AI
208299
- **openrouter** - OpenRouter (access to multiple models)
209300
- **bedrock** - AWS Bedrock (Claude, Llama, Titan, etc.)
301+
- **custom** - Custom OpenAI compatible AI provider, in-house solutions, LocalAI, Ollama, etc, just provide the url
210302

211303
### AI Configuration Options
212304

@@ -227,7 +319,8 @@ All configuration parameters are specified at the root level (all optional excep
227319
"additionalPromptContext": "...", // Additional prompt context
228320
"additionalSystemPromptContext": "...", // Additional system prompt context
229321
"log": false, // Enable logging
230-
"deploymentId": "..." // Azure OpenAI deployment ID (Azure only)
322+
"deploymentId": "...", // Azure OpenAI deployment ID (Azure only)
323+
"url": "..." // Custom OpenAI compatible AI provider url, in-house solutions, LocalAI, Ollama, etc
231324
}
232325
```
233326

@@ -463,8 +556,6 @@ For detailed information about configuring and using these integrations, see our
463556

464557
Integrations are currently in beta. Please report any issues to the [GitHub Test Reporter repository](https://github.com/ctrf-io/github-test-reporter/issues).
465558

466-
467-
468559
## Further Processing
469560

470561
You can further process the CTRF report by using the output `report` from the action or by using the `write-ctrf-to-file` input. For reports larger than 1MB, consider using the `write-ctrf-to-file` input.

docs/report-showcase.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -573,7 +573,7 @@ excluding any retries. This metric identifies tests with consistent failures,
573573
enabling teams to prioritize fixes and improve overall test reliability.
574574

575575
Use the `previous-results-max` input to state how many previous results to
576-
include in the calculation. The default is 10.
576+
include in the calculation.
577577

578578
This report goes nicely with the insights-report, flaky-rate-report, and
579579
slowest-report to provide a comprehensive view of the performance of your tests
@@ -664,7 +664,7 @@ initially but pass upon retry. Using test retries is essential for detecting
664664
flaky tests within CTRF.
665665

666666
Use the `previous-results-max` input to state how many previous results to
667-
include in the calculation. The default is 10.
667+
include in the calculation.
668668

669669
This report goes fits nicely with the insights-report, fail-rate-report, and
670670
slowest-report to provide a comprehensive view of the performance of your tests

images/ai-summary.png

168 KB
Loading

images/github.png

-52 KB
Loading

images/insights.png

-173 KB
Loading

images/pr-comment-with-fail.png

70.5 KB
Loading

images/pr-comment.png

43.1 KB
Loading

0 commit comments

Comments
 (0)