Skip to content

Documentation updates #251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ma11hewThomas merged 3 commits into from
Nov 23, 2025
Merged

Documentation updates #251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
728ec25
docs: refresh
Ma11hewThomas Nov 22, 2025
fc126c9
docs: update
Ma11hewThomas Nov 22, 2025
87745b0
docs: update
Ma11hewThomas Nov 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 23 additions & 123 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,31 +27,29 @@ You can support the project by:

**🧘 Super flexible:** start fast with powerful built-in reports or go fully custom with your own templates

**🤖 Continuous AI:** summarize test results with AI and get explanations of them with contextual insights
**🤖 AI summary:** summarize test results with AI and get explanations of them with contextual insights

**🔌 Framework-agnostic:** works with any testing tool

## Table of Contents

1. [Basic Usage](#basic-usage)
2. [Insights Usage](#insights-usage)
3. [Continuous AI Usage](#continuous-ai-usage)
4. [Pull Request Comment Usage](#pull-request-comment-usage)
5. [Super Flexible Usage](#super-flexible-usage)
6. [Report Showcase](#report-showcase)
7. [Generate a report](#generate-a-report)
8. [Available Inputs](#available-inputs)
9. [Generating an AI Report](#generating-an-ai-report)
10. [Pull Requests](#pull-requests)
11. [Status Checks](#status-checks)
12. [Build Your Own Report](#build-your-own-report)
13. [Customizing Report Order](#customizing-report-order)
14. [Community Reports](#community-reports)
15. [GitHub Token](#github-token)
16. [Storing Artifacts](#storing-artifacts)
17. [Filtering](#filtering)
18. [Integrations](#integrations)
19. [What is CTRF?](#what-is-ctrf)
3. [Pull Request Comment Usage](#pull-request-comment-usage)
4. [Super Flexible Usage](#super-flexible-usage)
5. [Report Showcase](#report-showcase)
6. [Generate a report](#generate-a-report)
7. [Available Inputs](#available-inputs)
8. [Pull Requests](#pull-requests)
9. [Status Checks](#status-checks)
10. [Build Your Own Report](#build-your-own-report)
11. [Customizing Report Order](#customizing-report-order)
12. [Community Reports](#community-reports)
13. [GitHub Token](#github-token)
14. [Storing Artifacts](#storing-artifacts)
15. [Filtering](#filtering)
16. [Integrations](#integrations)
17. [What is CTRF?](#what-is-ctrf)

## Basic Usage

Expand Down Expand Up @@ -98,40 +96,6 @@ This will create a job summary with insights on failures, flakiness and test dur
<img src="images/insights.png" alt="GitHub Test Reporter" width="600">
</div>

## Continuous AI Usage

To get started with continuous AI analysis of test results, add the following to your workflow file:

```yaml
- name: Publish Test Report with AI Analysis
uses: ctrf-io/github-test-reporter@v1
with:
report-path: './ctrf/*.json'
ai-summary-report: true
on-fail-only: true
summary: false
pull-request: true
ai: |
{
"provider": "openai",
"model": "gpt-5"
}
if: always()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

Add your AI API key to the repository secrets. Choose from a variety of AI providers and models. See [AI Configuration](#ai-configuration) for more information.

Make sure your GitHub Token has pull request write permission.

This will create a pull request comment with a summary of the AI analysis.

<div align="center">
<img src="images/ai-summary.png" alt="GitHub Test Reporter" width="600">
</div>

## Pull Request Comment Usage

To get started with pull request comments, add the following to your workflow file:
Expand Down Expand Up @@ -248,82 +212,12 @@ For more advanced usage, there are several inputs available.
group-by: 'filePath' # Specify grouping for applicable reports (e.g., suite or file path). Default is filePath
always-group-by: false # Force grouping by suite or file path for all reports. Default is false
report-order: 'summary-report,failed-report,flaky-report,skipped-report,test-report' # Comma-separated list of report types to specify the order in which reports should be displayed
ai: '{}' # JSON configuration for AI-powered test analysis. See AI Configuration section below
integrations-config: '{}' # JSON configuration for integrations with other developer tools
if: always()
```

Only `report-path` is required.

## Generating an AI Report

You can generate human-readable AI reports for your failed tests using models
from the leading AI providers. The GitHub Test Reporter now features dedicated
AI-first configuration for seamless integration with continuous AI workflows.

### AI Configuration

Use the `ai` input to configure AI-powered test analysis. Simply provide a JSON object
with the provider and any optional settings:

```yaml
- name: Publish Test Report with AI Analysis
uses: ctrf-io/github-test-reporter@v1
with:
report-path: './ctrf/*.json'
ai-summary-report: true
pull-request: true
ai: |
{
"provider": "openai",
"model": "gpt-4"
}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
if: always()
```

### Supported AI Providers

The following AI providers are supported:

- **openai** - OpenAI (GPT-4, GPT-3.5, etc.)
- **claude** - Anthropic Claude
- **gemini** - Google Gemini
- **azure-openai** - Azure OpenAI Service
- **grok** - xAI Grok
- **deepseek** - DeepSeek
- **mistral** - Mistral AI
- **perplexity** - Perplexity AI
- **openrouter** - OpenRouter (access to multiple models)
- **bedrock** - AWS Bedrock (Claude, Llama, Titan, etc.)
- **custom** - Custom OpenAI compatible AI provider, in-house solutions, LocalAI, Ollama, etc, just provide the url

### AI Configuration Options

All configuration parameters are specified at the root level (all optional except `provider`):

```json
{
"provider": "openai", // Required: AI provider to use
"model": "gpt-4", // AI model to use
"temperature": 0.7, // Creativity (0-2)
"maxTokens": 2000, // Max response length
"systemPrompt": "...", // Custom system prompt
"frequencyPenalty": 0, // Frequency penalty (0-2)
"presencePenalty": 0, // Presence penalty (0-2)
"topP": 1, // Nucleus sampling
"maxMessages": 10, // Max failed tests to analyze
"consolidate": true, // Consolidate multiple failures
"additionalPromptContext": "...", // Additional prompt context
"additionalSystemPromptContext": "...", // Additional system prompt context
"log": false, // Enable logging
"deploymentId": "...", // Azure OpenAI deployment ID (Azure only)
"url": "..." // Custom OpenAI compatible AI provider url, in-house solutions, LocalAI, Ollama, etc
}
```

## Pull Requests

You can add a pull request comment by using the `pull-request-report` input:
Expand Down Expand Up @@ -551,11 +445,17 @@ CTRF tooling offers seamless developer tool integration, allowing you to combine
|------------|-------------|------------|
| Slack Test Reporter | Send test results and notifications to Slack channels | [ctrf-io/slack-test-reporter](https://github.com/ctrf-io/slack-test-reporter) |
| Microsoft Teams Test Reporter | Post test results and alerts to Teams channels | [ctrf-io/teams-test-reporter](https://github.com/ctrf-io/teams-test-reporter) |
| AI Test Reporter | Intelligent test analysis using leading AI models | [ctrf-io/ai-test-reporter](https://github.com/ctrf-io/ai-test-reporter) |

For detailed information about configuring and using these integrations, see our [Integrations Documentation](docs/integrations.md).

Integrations are currently in beta. Please report any issues to the [GitHub Test Reporter repository](https://github.com/ctrf-io/github-test-reporter/issues).

## Generating an AI Report

You can generate human-readable AI report for your failed tests using models
from the leading AI providers by using the AI Test Reporter integration or the [AI Test Reporter](https://github.com/ctrf-io/ai-test-reporter) directly.

## Further Processing

You can further process the CTRF report by using the output `report` from the action or by using the `write-ctrf-to-file` input. For reports larger than 1MB, consider using the `write-ctrf-to-file` input.
Expand Down Expand Up @@ -589,4 +489,4 @@ analyzing test outcomes across multiple platforms becomes more straightforward.
## Support Us

If you find this project useful, consider giving it a GitHub star ⭐ It means a
lot to us.
lot to us.
167 changes: 152 additions & 15 deletions docs/build-your-own-report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,6 @@ This template demonstrates:

And what it looks like:

![Custom Report Example](../images/custom-one.png)

## Handlebars

Handlebars is a simple templating language that lets you insert data into your
Expand All @@ -107,7 +105,7 @@ When writing your template, you can use Handlebars helpers:
- `{{eq arg1 arg2}}`: Compares two arguments and returns true if they are equal.

See available helpers
[here](https://github.com/ctrf-io/handlebars-helpers-ctrf).
[handlebars-helpers-ctrf repository](https://github.com/ctrf-io/handlebars-helpers-ctrf).

We welcome contributions for additional helpers.

Expand Down Expand Up @@ -145,25 +143,164 @@ Example accessing test data:

## GitHub Properties

GitHub properties are made available to use in your template. You can access
these properties by using the `github` property, for example `github.repoName`
GitHub properties are made available to use in your template. You can access these properties by using the `github` property, for example `github.repoName` or `github.actor`.

### Root Context Properties

Access workflow and execution information:

**Workflow Information:**

- `github.workflow` - Name of the workflow
- `github.action` / `github.action_name` - Current action name
- `github.job` / `github.job_id` / `github.jobName` - Current job identifier
- `github.runNumber` / `github.run_number` - Unique number for each workflow run
- `github.runId` / `github.run_id` / `github.workflowId` - Unique identifier for the workflow run
- `github.workflowName` - Workflow name (legacy property)

**Event Information:**

- `github.eventName` / `github.event_name` - Event that triggered the workflow (e.g., "push", "pull_request")
- `github.actor` / `github.actor_name` / `github.actorName` - User who triggered the workflow
- `github.sha` - Commit SHA that triggered the workflow
- `github.ref` - Git ref that triggered the workflow (e.g., "refs/heads/main")
- `github.branchName` - Current branch name

**Repository & URLs:**

- `github.repoName` - Repository name
- `github.buildUrl` / `github.build_url` - URL to the workflow run summary
- `github.serverUrl` / `github.server_url` / `github.baseURL` - GitHub server URL (e.g., `https://github.com`)
- `github.apiUrl` / `github.api_url` - GitHub API URL (e.g., `https://api.github.com`)
- `github.graphqlUrl` / `github.graphql_url` - GitHub GraphQL API URL

**Pull Request (at root level):**

- `github.pullRequestNumber` - PR number (legacy property, prefer `github.pullRequest.number`)

### Repository Properties (`github.repository`)

Access repository information:

**Basic Info:**

- `github.repository.name` - Repository name
- `github.repository.fullName` / `github.repository.full_name` - Owner and repository name (e.g., "owner/repo")
- `github.repository.description` - Repository description
- `github.repository.language` - Primary language
- `github.repository.defaultBranch` / `github.repository.default_branch` - Default branch name
- `github.repository.licenseName` / `github.repository.license_name` - License name

**Statistics:**

- `github.repository.size` - Repository size in KB
- `github.repository.stargazersCount` / `github.repository.stargazers_count` - Number of stars
- `github.repository.openIssuesCount` / `github.repository.open_issues_count` - Number of open issues

**URLs:**

- `github.repository.htmlUrl` / `github.repository.html_url` - Repository URL
- `github.repository.cloneUrl` / `github.repository.clone_url` - HTTPS clone URL
- `github.repository.sshUrl` / `github.repository.ssh_url` - SSH clone URL
- `github.repository.compareUrl` / `github.repository.compare_url` - Compare URL
- `github.repository.contributorsUrl` / `github.repository.contributors_url` - Contributors page URL
- `github.repository.deploymentsUrl` / `github.repository.deployments_url` - Deployments URL
- `github.repository.downloadsUrl` / `github.repository.downloads_url` - Downloads URL
- `github.repository.eventsUrl` / `github.repository.events_url` - Events URL
- `github.repository.forksUrl` / `github.repository.forks_url` - Forks URL
- `github.repository.stargazersUrl` / `github.repository.stargazers_url` - Stargazers URL
- `github.repository.statusesUrl` / `github.repository.statuses_url` - Commit statuses URL
- `github.repository.subscriptionUrl` / `github.repository.subscription_url` - Subscription URL
- `github.repository.tagsUrl` / `github.repository.tags_url` - Tags URL
- `github.repository.teamsUrl` / `github.repository.teams_url` - Teams URL

**Settings:**

- `github.repository.allowForking` / `github.repository.allow_forking` - Whether forking is allowed
- `github.repository.createdAt` / `github.repository.created_at` - Repository creation timestamp

### Pull Request Properties (`github.pullRequest`)

You can access the entire context via the `github.context` property.
Access pull request information (available when triggered by PR events):

[Contexts](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs)
are a way to access information about workflow runs, variables, runner
environments, jobs, and steps. Each context is an object that contains
properties, which can be strings or other objects.
**Basic Info:**

Contexts, objects, and properties will vary significantly under different
workflow run conditions. For example, the matrix context is only populated for
jobs in a matrix.
- `github.pullRequest.id` - PR ID
- `github.pullRequest.number` - PR number
- `github.pullRequest.title` - PR title
- `github.pullRequest.body` - PR description
- `github.pullRequest.state` - PR state ("open", "closed")
- `github.pullRequest.draft` - Whether PR is a draft (boolean)
- `github.pullRequest.rebaseable` - Whether PR can be rebased (boolean or null)

You can see the content of the context by printing it in the logs:
**Changes:**

- `github.pullRequest.additions` - Lines added
- `github.pullRequest.deletions` - Lines deleted
- `github.pullRequest.changedFiles` / `github.pullRequest.changed_files` - Number of files changed

**Collaboration:**

- `github.pullRequest.assignee` - Assigned user object (or null)
- `github.pullRequest.assignees` - Array of assigned users
- `github.pullRequest.authorAssociation` / `github.pullRequest.author_association` - Author's association with the repository
- `github.pullRequest.requestedReviewers` / `github.pullRequest.requested_reviewers` - Array of requested reviewers
- `github.pullRequest.requestedTeams` / `github.pullRequest.requested_teams` - Array of requested teams
- `github.pullRequest.comments` - Number of comments
- `github.pullRequest.reviewComments` / `github.pullRequest.review_comments` - Number of review comments
- `github.pullRequest.labels` - Array of labels

**URLs:**

- `github.pullRequest.htmlUrl` / `github.pullRequest.html_url` - PR URL
- `github.pullRequest.diffUrl` / `github.pullRequest.diff_url` - Diff URL
- `github.pullRequest.patchUrl` / `github.pullRequest.patch_url` - Patch URL

**Timestamps:**

- `github.pullRequest.createdAt` / `github.pullRequest.created_at` - Creation timestamp
- `github.pullRequest.closedAt` / `github.pullRequest.closed_at` - Closed timestamp (or null)
- `github.pullRequest.pushedAt` / `github.pullRequest.pushed_at` - Last push timestamp

**Merge Settings:**

- `github.pullRequest.autoMerge` / `github.pullRequest.auto_merge` - Auto-merge configuration (or null)

### Sender Properties (`github.sender`)

Access information about the user who triggered the workflow:

- `github.sender.login` - Username
- `github.sender.id` - User ID
- `github.sender.nodeId` / `github.sender.node_id` - Node ID
- `github.sender.type` - User type (e.g., "User", "Bot")
- `github.sender.siteAdmin` / `github.sender.site_admin` - Whether user is a site admin (boolean)
- `github.sender.htmlUrl` / `github.sender.html_url` - User profile URL
- `github.sender.avatarUrl` / `github.sender.avatar_url` - User avatar URL
- `github.sender.gravatarId` / `github.sender.gravatar_id` - Gravatar ID

### Full Context Access

You can access the complete webhook event payload via `github.context`. This contains the raw event payload data sent by GitHub when the workflow was triggered (e.g., the full `pull_request` object, `repository` object, `issue` object, etc. depending on the event type).

This is useful for accessing event-specific data that isn't exposed in the structured properties above. For example:

- `github.context.pull_request.head.ref` - Head branch name
- `github.context.pull_request.base.ref` - Base branch name
- `github.context.issue` - Issue data (for issue events)
- `github.context.comment` - Comment data (for comment events)

See the [GitHub Actions events documentation](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows) for available event types.

**Note:** Contexts, objects, and properties will vary significantly under different workflow run conditions. For example, pull request properties are only available when the workflow is triggered by a pull request event.

### Debugging Available Properties

To see all available properties for your specific workflow, print the context in your workflow logs:

```yaml
- name: Print GitHub Context
env:
CONTEXT: ${{ toJson(github) }}
run: echo "$CONTEXT" | jq .
```
```
Loading
Loading