diff --git a/README.md b/README.md index 82601dd6..9253ac33 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ You can support the project by: **🧘 Super flexible:** start fast with powerful built-in reports or go fully custom with your own templates -**🤖 Continuous AI:** summarize test results with AI and get explanations of them with contextual insights +**🤖 AI summary:** summarize test results with AI and get explanations of them with contextual insights **🔌 Framework-agnostic:** works with any testing tool @@ -35,23 +35,21 @@ You can support the project by: 1. [Basic Usage](#basic-usage) 2. [Insights Usage](#insights-usage) -3. [Continuous AI Usage](#continuous-ai-usage) -4. [Pull Request Comment Usage](#pull-request-comment-usage) -5. [Super Flexible Usage](#super-flexible-usage) -6. [Report Showcase](#report-showcase) -7. [Generate a report](#generate-a-report) -8. [Available Inputs](#available-inputs) -9. [Generating an AI Report](#generating-an-ai-report) -10. [Pull Requests](#pull-requests) -11. [Status Checks](#status-checks) -12. [Build Your Own Report](#build-your-own-report) -13. [Customizing Report Order](#customizing-report-order) -14. [Community Reports](#community-reports) -15. [GitHub Token](#github-token) -16. [Storing Artifacts](#storing-artifacts) -17. [Filtering](#filtering) -18. [Integrations](#integrations) -19. [What is CTRF?](#what-is-ctrf) +3. [Pull Request Comment Usage](#pull-request-comment-usage) +4. [Super Flexible Usage](#super-flexible-usage) +5. [Report Showcase](#report-showcase) +6. [Generate a report](#generate-a-report) +7. [Available Inputs](#available-inputs) +8. [Pull Requests](#pull-requests) +9. [Status Checks](#status-checks) +10. [Build Your Own Report](#build-your-own-report) +11. [Customizing Report Order](#customizing-report-order) +12. [Community Reports](#community-reports) +13. [GitHub Token](#github-token) +14. [Storing Artifacts](#storing-artifacts) +15. [Filtering](#filtering) +16. [Integrations](#integrations) +17. [What is CTRF?](#what-is-ctrf) ## Basic Usage @@ -98,40 +96,6 @@ This will create a job summary with insights on failures, flakiness and test dur GitHub Test Reporter -## Continuous AI Usage - -To get started with continuous AI analysis of test results, add the following to your workflow file: - -```yaml -- name: Publish Test Report with AI Analysis - uses: ctrf-io/github-test-reporter@v1 - with: - report-path: './ctrf/*.json' - ai-summary-report: true - on-fail-only: true - summary: false - pull-request: true - ai: | - { - "provider": "openai", - "model": "gpt-5" - } - if: always() - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} -``` - -Add your AI API key to the repository secrets. Choose from a variety of AI providers and models. See [AI Configuration](#ai-configuration) for more information. - -Make sure your GitHub Token has pull request write permission. - -This will create a pull request comment with a summary of the AI analysis. - -
-GitHub Test Reporter -
- ## Pull Request Comment Usage To get started with pull request comments, add the following to your workflow file: @@ -248,82 +212,12 @@ For more advanced usage, there are several inputs available. group-by: 'filePath' # Specify grouping for applicable reports (e.g., suite or file path). Default is filePath always-group-by: false # Force grouping by suite or file path for all reports. Default is false report-order: 'summary-report,failed-report,flaky-report,skipped-report,test-report' # Comma-separated list of report types to specify the order in which reports should be displayed - ai: '{}' # JSON configuration for AI-powered test analysis. See AI Configuration section below integrations-config: '{}' # JSON configuration for integrations with other developer tools if: always() ``` Only `report-path` is required. -## Generating an AI Report - -You can generate human-readable AI reports for your failed tests using models -from the leading AI providers. The GitHub Test Reporter now features dedicated -AI-first configuration for seamless integration with continuous AI workflows. - -### AI Configuration - -Use the `ai` input to configure AI-powered test analysis. Simply provide a JSON object -with the provider and any optional settings: - -```yaml -- name: Publish Test Report with AI Analysis - uses: ctrf-io/github-test-reporter@v1 - with: - report-path: './ctrf/*.json' - ai-summary-report: true - pull-request: true - ai: | - { - "provider": "openai", - "model": "gpt-4" - } - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - if: always() -``` - -### Supported AI Providers - -The following AI providers are supported: - -- **openai** - OpenAI (GPT-4, GPT-3.5, etc.) -- **claude** - Anthropic Claude -- **gemini** - Google Gemini -- **azure-openai** - Azure OpenAI Service -- **grok** - xAI Grok -- **deepseek** - DeepSeek -- **mistral** - Mistral AI -- **perplexity** - Perplexity AI -- **openrouter** - OpenRouter (access to multiple models) -- **bedrock** - AWS Bedrock (Claude, Llama, Titan, etc.) -- **custom** - Custom OpenAI compatible AI provider, in-house solutions, LocalAI, Ollama, etc, just provide the url - -### AI Configuration Options - -All configuration parameters are specified at the root level (all optional except `provider`): - -```json -{ - "provider": "openai", // Required: AI provider to use - "model": "gpt-4", // AI model to use - "temperature": 0.7, // Creativity (0-2) - "maxTokens": 2000, // Max response length - "systemPrompt": "...", // Custom system prompt - "frequencyPenalty": 0, // Frequency penalty (0-2) - "presencePenalty": 0, // Presence penalty (0-2) - "topP": 1, // Nucleus sampling - "maxMessages": 10, // Max failed tests to analyze - "consolidate": true, // Consolidate multiple failures - "additionalPromptContext": "...", // Additional prompt context - "additionalSystemPromptContext": "...", // Additional system prompt context - "log": false, // Enable logging - "deploymentId": "...", // Azure OpenAI deployment ID (Azure only) - "url": "..." // Custom OpenAI compatible AI provider url, in-house solutions, LocalAI, Ollama, etc -} -``` - ## Pull Requests You can add a pull request comment by using the `pull-request-report` input: @@ -551,11 +445,17 @@ CTRF tooling offers seamless developer tool integration, allowing you to combine |------------|-------------|------------| | Slack Test Reporter | Send test results and notifications to Slack channels | [ctrf-io/slack-test-reporter](https://github.com/ctrf-io/slack-test-reporter) | | Microsoft Teams Test Reporter | Post test results and alerts to Teams channels | [ctrf-io/teams-test-reporter](https://github.com/ctrf-io/teams-test-reporter) | +| AI Test Reporter | Intelligent test analysis using leading AI models | [ctrf-io/ai-test-reporter](https://github.com/ctrf-io/ai-test-reporter) | For detailed information about configuring and using these integrations, see our [Integrations Documentation](docs/integrations.md). Integrations are currently in beta. Please report any issues to the [GitHub Test Reporter repository](https://github.com/ctrf-io/github-test-reporter/issues). +## Generating an AI Report + +You can generate human-readable AI report for your failed tests using models +from the leading AI providers by using the AI Test Reporter integration or the [AI Test Reporter](https://github.com/ctrf-io/ai-test-reporter) directly. + ## Further Processing You can further process the CTRF report by using the output `report` from the action or by using the `write-ctrf-to-file` input. For reports larger than 1MB, consider using the `write-ctrf-to-file` input. @@ -589,4 +489,4 @@ analyzing test outcomes across multiple platforms becomes more straightforward. ## Support Us If you find this project useful, consider giving it a GitHub star ⭐ It means a -lot to us. \ No newline at end of file +lot to us. diff --git a/docs/build-your-own-report.md b/docs/build-your-own-report.md index 571fd6c6..b550b9cd 100644 --- a/docs/build-your-own-report.md +++ b/docs/build-your-own-report.md @@ -92,8 +92,6 @@ This template demonstrates: And what it looks like: -![Custom Report Example](../images/custom-one.png) - ## Handlebars Handlebars is a simple templating language that lets you insert data into your @@ -107,7 +105,7 @@ When writing your template, you can use Handlebars helpers: - `{{eq arg1 arg2}}`: Compares two arguments and returns true if they are equal. See available helpers -[here](https://github.com/ctrf-io/handlebars-helpers-ctrf). +[handlebars-helpers-ctrf repository](https://github.com/ctrf-io/handlebars-helpers-ctrf). We welcome contributions for additional helpers. @@ -145,25 +143,164 @@ Example accessing test data: ## GitHub Properties -GitHub properties are made available to use in your template. You can access -these properties by using the `github` property, for example `github.repoName` +GitHub properties are made available to use in your template. You can access these properties by using the `github` property, for example `github.repoName` or `github.actor`. + +### Root Context Properties + +Access workflow and execution information: + +**Workflow Information:** + +- `github.workflow` - Name of the workflow +- `github.action` / `github.action_name` - Current action name +- `github.job` / `github.job_id` / `github.jobName` - Current job identifier +- `github.runNumber` / `github.run_number` - Unique number for each workflow run +- `github.runId` / `github.run_id` / `github.workflowId` - Unique identifier for the workflow run +- `github.workflowName` - Workflow name (legacy property) + +**Event Information:** + +- `github.eventName` / `github.event_name` - Event that triggered the workflow (e.g., "push", "pull_request") +- `github.actor` / `github.actor_name` / `github.actorName` - User who triggered the workflow +- `github.sha` - Commit SHA that triggered the workflow +- `github.ref` - Git ref that triggered the workflow (e.g., "refs/heads/main") +- `github.branchName` - Current branch name + +**Repository & URLs:** + +- `github.repoName` - Repository name +- `github.buildUrl` / `github.build_url` - URL to the workflow run summary +- `github.serverUrl` / `github.server_url` / `github.baseURL` - GitHub server URL (e.g., `https://github.com`) +- `github.apiUrl` / `github.api_url` - GitHub API URL (e.g., `https://api.github.com`) +- `github.graphqlUrl` / `github.graphql_url` - GitHub GraphQL API URL + +**Pull Request (at root level):** + +- `github.pullRequestNumber` - PR number (legacy property, prefer `github.pullRequest.number`) + +### Repository Properties (`github.repository`) + +Access repository information: + +**Basic Info:** + +- `github.repository.name` - Repository name +- `github.repository.fullName` / `github.repository.full_name` - Owner and repository name (e.g., "owner/repo") +- `github.repository.description` - Repository description +- `github.repository.language` - Primary language +- `github.repository.defaultBranch` / `github.repository.default_branch` - Default branch name +- `github.repository.licenseName` / `github.repository.license_name` - License name + +**Statistics:** + +- `github.repository.size` - Repository size in KB +- `github.repository.stargazersCount` / `github.repository.stargazers_count` - Number of stars +- `github.repository.openIssuesCount` / `github.repository.open_issues_count` - Number of open issues + +**URLs:** + +- `github.repository.htmlUrl` / `github.repository.html_url` - Repository URL +- `github.repository.cloneUrl` / `github.repository.clone_url` - HTTPS clone URL +- `github.repository.sshUrl` / `github.repository.ssh_url` - SSH clone URL +- `github.repository.compareUrl` / `github.repository.compare_url` - Compare URL +- `github.repository.contributorsUrl` / `github.repository.contributors_url` - Contributors page URL +- `github.repository.deploymentsUrl` / `github.repository.deployments_url` - Deployments URL +- `github.repository.downloadsUrl` / `github.repository.downloads_url` - Downloads URL +- `github.repository.eventsUrl` / `github.repository.events_url` - Events URL +- `github.repository.forksUrl` / `github.repository.forks_url` - Forks URL +- `github.repository.stargazersUrl` / `github.repository.stargazers_url` - Stargazers URL +- `github.repository.statusesUrl` / `github.repository.statuses_url` - Commit statuses URL +- `github.repository.subscriptionUrl` / `github.repository.subscription_url` - Subscription URL +- `github.repository.tagsUrl` / `github.repository.tags_url` - Tags URL +- `github.repository.teamsUrl` / `github.repository.teams_url` - Teams URL + +**Settings:** + +- `github.repository.allowForking` / `github.repository.allow_forking` - Whether forking is allowed +- `github.repository.createdAt` / `github.repository.created_at` - Repository creation timestamp + +### Pull Request Properties (`github.pullRequest`) -You can access the entire context via the `github.context` property. +Access pull request information (available when triggered by PR events): -[Contexts](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs) -are a way to access information about workflow runs, variables, runner -environments, jobs, and steps. Each context is an object that contains -properties, which can be strings or other objects. +**Basic Info:** -Contexts, objects, and properties will vary significantly under different -workflow run conditions. For example, the matrix context is only populated for -jobs in a matrix. +- `github.pullRequest.id` - PR ID +- `github.pullRequest.number` - PR number +- `github.pullRequest.title` - PR title +- `github.pullRequest.body` - PR description +- `github.pullRequest.state` - PR state ("open", "closed") +- `github.pullRequest.draft` - Whether PR is a draft (boolean) +- `github.pullRequest.rebaseable` - Whether PR can be rebased (boolean or null) -You can see the content of the context by printing it in the logs: +**Changes:** + +- `github.pullRequest.additions` - Lines added +- `github.pullRequest.deletions` - Lines deleted +- `github.pullRequest.changedFiles` / `github.pullRequest.changed_files` - Number of files changed + +**Collaboration:** + +- `github.pullRequest.assignee` - Assigned user object (or null) +- `github.pullRequest.assignees` - Array of assigned users +- `github.pullRequest.authorAssociation` / `github.pullRequest.author_association` - Author's association with the repository +- `github.pullRequest.requestedReviewers` / `github.pullRequest.requested_reviewers` - Array of requested reviewers +- `github.pullRequest.requestedTeams` / `github.pullRequest.requested_teams` - Array of requested teams +- `github.pullRequest.comments` - Number of comments +- `github.pullRequest.reviewComments` / `github.pullRequest.review_comments` - Number of review comments +- `github.pullRequest.labels` - Array of labels + +**URLs:** + +- `github.pullRequest.htmlUrl` / `github.pullRequest.html_url` - PR URL +- `github.pullRequest.diffUrl` / `github.pullRequest.diff_url` - Diff URL +- `github.pullRequest.patchUrl` / `github.pullRequest.patch_url` - Patch URL + +**Timestamps:** + +- `github.pullRequest.createdAt` / `github.pullRequest.created_at` - Creation timestamp +- `github.pullRequest.closedAt` / `github.pullRequest.closed_at` - Closed timestamp (or null) +- `github.pullRequest.pushedAt` / `github.pullRequest.pushed_at` - Last push timestamp + +**Merge Settings:** + +- `github.pullRequest.autoMerge` / `github.pullRequest.auto_merge` - Auto-merge configuration (or null) + +### Sender Properties (`github.sender`) + +Access information about the user who triggered the workflow: + +- `github.sender.login` - Username +- `github.sender.id` - User ID +- `github.sender.nodeId` / `github.sender.node_id` - Node ID +- `github.sender.type` - User type (e.g., "User", "Bot") +- `github.sender.siteAdmin` / `github.sender.site_admin` - Whether user is a site admin (boolean) +- `github.sender.htmlUrl` / `github.sender.html_url` - User profile URL +- `github.sender.avatarUrl` / `github.sender.avatar_url` - User avatar URL +- `github.sender.gravatarId` / `github.sender.gravatar_id` - Gravatar ID + +### Full Context Access + +You can access the complete webhook event payload via `github.context`. This contains the raw event payload data sent by GitHub when the workflow was triggered (e.g., the full `pull_request` object, `repository` object, `issue` object, etc. depending on the event type). + +This is useful for accessing event-specific data that isn't exposed in the structured properties above. For example: + +- `github.context.pull_request.head.ref` - Head branch name +- `github.context.pull_request.base.ref` - Base branch name +- `github.context.issue` - Issue data (for issue events) +- `github.context.comment` - Comment data (for comment events) + +See the [GitHub Actions events documentation](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows) for available event types. + +**Note:** Contexts, objects, and properties will vary significantly under different workflow run conditions. For example, pull request properties are only available when the workflow is triggered by a pull request event. + +### Debugging Available Properties + +To see all available properties for your specific workflow, print the context in your workflow logs: ```yaml - name: Print GitHub Context env: CONTEXT: ${{ toJson(github) }} run: echo "$CONTEXT" | jq . -``` \ No newline at end of file +``` diff --git a/docs/fork-pull-requests.md b/docs/fork-pull-requests.md index 92ac21cc..990eeead 100644 --- a/docs/fork-pull-requests.md +++ b/docs/fork-pull-requests.md @@ -1,6 +1,6 @@ -# 🚀 GitHub Actions: Securely Commenting Test Results on All PRs +# Safely Commenting on Forked PRs -## 📌 Overview +## Overview This tutorial explains how to set up a GitHub Actions workflow that runs tests and comments on the results of pull requests (PRs), including those from forks. Using two workflows ensures security while allowing test reports to be posted on all types of PRs. @@ -11,15 +11,7 @@ The setup consists of two workflows: This method is applicable to any project using GitHub Actions for CI/CD, ensuring a secure and efficient way to handle test reporting. ---- - -## ⚠️ Important Note - -These workflows should be implemented on the default branch of the repository (either `master` or `main` in newer repositories) to ensure proper execution and integration. Running workflows on other branches may lead to unexpected behavior, security issues, or failure to post comments on pull requests. - ---- - -## 🔐 Why Use Two Workflows? +## Why Use Two Workflows? GitHub restricts workflows triggered by `pull_request` events from writing to the base repository when PRs originate from forks. This limitation prevents workflows from commenting on pull requests directly. @@ -32,128 +24,115 @@ To mitigate this, we split the workflow into two: This method ensures that test results are always accessible while maintaining security. ---- - -## 🔄 Step-by-Step Workflow Execution +## Step-by-Step Workflow Execution -### **1️⃣ workflowA: Running Tests and Uploading Artifacts** +### **workflowA: Running Tests and Uploading Artifacts** -This workflow is triggered when a pull request is opened, synchronized, or reopened on any branch. It performs the following steps: - -- **Check out PR code**: - -```yaml -- name: Check out PR code - uses: actions/checkout@v4 -``` - -- **Run Tests**: - -```yaml -- name: Run Tests - run: | - ./run-tests.sh # Replace with your actual test command -``` - -- **Convert Test Results to a Compatible Format**: - -```yaml -- name: Convert Test Results - run: | - npx test-result-converter ./testReport.xml -o ./results/test-report.json # Modify based on your test framework -``` +This workflow is triggered when a pull request is opened, synchronized, or reopened on any branch. -- **Upload Test Report Artifact**: +**Complete workflow file (`.github/workflows/test.yml`):** ```yaml -- name: Upload Test Report Artifact - uses: actions/upload-artifact@v4 - with: - name: testReport - path: ./results/test-report.json -``` - -- **Save PR Number and Upload as an Artifact**: - -To ensure that `workflowB` can correctly comment on the corresponding pull request, we save the PR number as an artifact in `workflowA`. Since `workflowB` is triggered by `workflowA` using `workflow_run`, it does not have direct access to the PR metadata. Uploading the PR number as an artifact allows `workflowB` to retrieve and use it for posting test results in the correct pull request. - -```yaml -- name: Save PR Number - run: echo "PR_NUMBER=${{ github.event.pull_request.number }}" >> $GITHUB_ENV - -- name: Upload PR Number as Artifact - run: echo $PR_NUMBER > pr_number.txt - shell: bash - -- name: Upload PR Number Artifact - uses: actions/upload-artifact@v4 - with: - name: pr_number - path: pr_number.txt +name: Run Tests + +on: + pull_request: + branches: ['**'] + +jobs: + test: + runs-on: ubuntu-latest + + steps: + - name: Check out PR code + uses: actions/checkout@v4 + + - name: Run Tests + run: | + ./run-tests.sh # Replace with your actual test command + + - name: Upload Test Report Artifact + uses: actions/upload-artifact@v4 + with: + name: testReport + path: ./results/ctrf-report.json ``` Since this workflow only requires read permissions, it avoids potential security risks when dealing with external contributions from forked repositories. The second workflow, which has the necessary permissions to write, is responsible for retrieving and posting the results, ensuring a secure and controlled execution process. ---- - -### **2️⃣ workflowB: Downloading Artifacts and Posting Results** - -This workflow is triggered when `workflowA` completes successfully. Since GitHub Actions does not allow direct artifact downloads across workflows using `actions/download-artifact`. - -- **Download Test Report Artifact:** Since GitHub Actions does not allow direct artifact downloads across workflows using `actions/download-artifact`, we use [`dawidd6/action-download-artifact@v8`](https://github.com/dawidd6/action-download-artifact) instead. This repository enables downloading artifacts from a previous workflow run by specifying the `run_id`, which is essential when handling artifacts between separate workflows. It follows these steps: -```yaml -- name: Download Test Report Artifact - uses: dawidd6/action-download-artifact@v8 - with: - name: testReport - run_id: ${{ github.event.workflow_run.id }} - path: artifacts -``` +### **workflowB: Downloading Artifacts and Posting Results** -- **Download PR Number Artifact**: - -```yaml -- name: Download PR Number Artifact - uses: dawidd6/action-download-artifact@v8 - with: - name: pr_number - run_id: ${{ github.event.workflow_run.id }} - path: pr_number -``` +This workflow is triggered when `workflowA` completes successfully. -- **Read PR Number**: +**Complete workflow file (`.github/workflows/report.yml`):** ```yaml -- name: Read PR Number - id: read_pr_number - run: | - PR_NUMBER=$(cat pr_number/pr_number.txt) - echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV +name: Publish Test Report + +on: + workflow_run: + workflows: ['Run Tests'] + types: [completed] + +permissions: + pull-requests: write + contents: read + +jobs: + report: + runs-on: ubuntu-latest + if: ${{ github.event.workflow_run.conclusion == 'success' }} + + steps: + - name: Download Test Report Artifact + uses: dawidd6/action-download-artifact@v8 + with: + name: testReport + run_id: ${{ github.event.workflow_run.id }} + path: artifacts + + - name: Determine PR number securely + id: get_pr + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + HEAD_SHA="${{ github.event.workflow_run.head_sha }}" + + PR_NUM=$(gh pr list \ + --state open \ + --json number,headRefOid \ + --jq ".[] | select(.headRefOid==\"${HEAD_SHA}\") | .number") + + if [ -z "$PR_NUM" ]; then + echo "No open PR found for head SHA ${HEAD_SHA}" + exit 1 + fi + + echo "PR_NUMBER=$PR_NUM" >> $GITHUB_ENV + + - name: Publish Test Report + uses: ctrf-io/github-test-reporter@v1 + with: + report-path: 'artifacts/ctrf-report.json' + pull-request: ${{ env.PR_NUMBER }} + update-comment: true + comment-tag: test-report + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` -- **Publish Test Report**: - -```yaml -- name: Publish Test Report - uses: test-reporter/github-test-reporter@v1.0.6 - with: - report-path: 'artifacts/test-report.json' - issue: ${{ env.PR_NUMBER }} - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} -``` +### Key Security Details -This final step posts the test report as a comment on the pull request, making it easy for contributors and maintainers to review test results. +> ⚠️ **Security Principle**: Never trust data from artifacts created by fork PRs for control flow decisions. Always retrieve critical metadata (like PR numbers) from trusted sources like GitHub's API. ---- +**Important points:** -## ✅ Conclusion +- **Download Test Report Artifact:** Since GitHub Actions does not allow direct artifact downloads across workflows using `actions/download-artifact`, we use [`dawidd6/action-download-artifact@v8`](https://github.com/dawidd6/action-download-artifact) instead. This enables downloading artifacts from a previous workflow run by specifying the `run_id`. -By structuring the workflows this way, we achieve the following: +- **Only download the test report artifact.** Do not download PR number or other metadata from the forked workflow. Artifacts from fork PRs are untrusted and should only be treated as test data. -- **Secure execution** without exposing repository write access to forked pull requests. -- **Successful test execution** and result upload. -- **Seamless commenting** on pull requests with test results while mitigating security risks. +- **Permissions:** The workflow explicitly requests `pull-requests: write` permission to comment on PRs, while limiting other permissions to `contents: read` for security. -This method ensures that test results are reliably posted while maintaining a secure GitHub Actions setup. Additionally, this approach scales effectively for large repositories with many PRs, as the artifact-based workflow minimizes redundant computations and ensures efficient resource utilization. 🚀 +- **Workflow completion check:** The job only runs if the previous workflow completed successfully (`github.event.workflow_run.conclusion == 'success'`), as shown in the complete workflow file above. +- **Determine PR Number:** Get the PR number via GitHub's API using the `head_sha` from `workflow_run`. This ensures the PR number comes from a trusted source. **Why this is secure:** The PR number comes from GitHub's trusted API, not from a forked PR's artifact. This eliminates the risk of a malicious contributor making the workflow comment on the wrong PR. \ No newline at end of file diff --git a/docs/integrations.md b/docs/integrations.md index 648f4692..71619b3d 100644 --- a/docs/integrations.md +++ b/docs/integrations.md @@ -10,6 +10,7 @@ Integrations are currently in beta. Please report any issues to the [GitHub Test |------------|-------------|------------| | Slack Test Reporter | Send test results and notifications to Slack channels | [ctrf-io/slack-test-reporter](https://github.com/ctrf-io/slack-test-reporter) | | Microsoft Teams Test Reporter | Post test results and alerts to Teams channels | [ctrf-io/teams-test-reporter](https://github.com/ctrf-io/teams-test-reporter) | +| AI Test Reporter | Intelligent test analysis using leading AI models | [ctrf-io/ai-test-reporter](https://github.com/ctrf-io/ai-test-reporter) | | JUnit to CTRF | Convert JUnit test results to CTRF format | [ctrf-io/junit-to-ctrf](https://github.com/ctrf-io/junit-to-ctrf) | To suggest a new integration, please open a discussion on the [CTRF Discussions](https://github.com/ctrf-io/ctrf/discussions). @@ -32,6 +33,10 @@ Integrations are configured using the `integrations-config` input in your GitHub "teams": { "enabled": true, "action": "results" + }, + "ai": { + "enabled": true, + "action": "openai" } } env: @@ -122,6 +127,8 @@ Requires the `TEAMS_WEBHOOK_URL` environment variable to be set. See the [Microsoft Teams Test Reporter](https://github.com/ctrf-io/teams-test-reporter) for more information. +### AI Integration + The AI integration provides intelligent analysis of your test results using advanced AI models. The AI integration is powered by the [AI Test Reporter](https://github.com/ctrf-io/ai-test-reporter), see the latestdocumentation for more information. @@ -204,4 +211,3 @@ Actions available: ``` See the [JUnit to CTRF](https://github.com/ctrf-io/junit-to-ctrf) for more information. - diff --git a/docs/report-showcase.md b/docs/report-showcase.md index 90fce97a..d8f87501 100644 --- a/docs/report-showcase.md +++ b/docs/report-showcase.md @@ -9,7 +9,6 @@ execution. - [Summary Report](#summary-report) - [Summary Delta Report](#summary-delta-report) - [GitHub Report](#github-report) - - [AI Summary Report](#ai-summary-report) - [AI Report](#ai-report) - [Test Report](#test-report) - [Test List Report](#test-list-report) @@ -270,65 +269,6 @@ Set the `github-report` input to true in your workflow configuration: --- ![GitHub Report](../images/github.png) -![GitHub Report](../images/github-failed.png) - -## AI Summary Report - -### Overview - -Provides an AI-generated comprehensive summary analysis of test failures across your entire test suite. Unlike the AI Report which analyzes individual failed tests, the AI Summary Report examines patterns and trends across all failures to identify systemic issues, code problems, timeout issues, and application-level concerns. This high-level analysis helps teams understand the root causes of multiple failures and prioritize fixes more effectively. - -The report includes: -- **Summary**: High-level overview of the failure patterns -- **Code Issues**: Specific code-level problems identified across failures -- **Timeout Issues**: Analysis of timing-related problems -- **Application Issues**: Broader application or environmental concerns -- **Recommendations**: Actionable suggestions to resolve the identified issues - -This report is especially useful for getting a bird's-eye view of test suite health and understanding whether failures are isolated incidents or symptoms of larger systemic issues. - -### Usage - -Set the `ai-summary-report` input to true in your workflow configuration: - -```yaml -- name: Publish Test Report - uses: ctrf-io/github-test-reporter@v1 - with: - report-path: './ctrf/*.json' - ai-summary-report: true - if: always() -``` - ---- - -### ✨ AI Summary - -**📋 Summary** - -Three related test failures in the `addFooterDisplayFlags` function reveal inconsistent logic when handling the `includeFlakyReportAllFooter` flag across different flaky test scenarios with previous suite results. Two tests expect the flag to be `false` but receive `true`, while one expects `true` but receives `false`. These are not intermittent flakiness issues but consistent logic errors that have affected approximately 27% of test runs. - -**🐛 Code Issues** - -• The **addFooterDisplayFlags** function contains contradictory or inverted conditional logic when evaluating whether to set `includeFlakyReportAllFooter` based on flaky test presence across runs and previous results. The function appears to be setting the flag to the opposite of the expected value in multiple scenarios involving flaky test detection with previous suite results. - -• Logic for determining when flaky tests exist "across all runs" versus when they don't is either inverted or missing proper condition checks, causing the flag to be enabled when it should be disabled and vice versa in different test scenarios. - -• The combined scenario handling (flaky tests in current AND across all runs) is incorrectly evaluating conditions when merging current results with previous historical data, failing to properly suppress the footer flag when flaky tests are detected. - -**⚠️ Application Issues** - -• The test suite shows a consistent 27% failure rate across 52 runs for these specific flag-setting scenarios, indicating a persistent, reproducible bug rather than environmental or timing-related flakiness. - -**💡 Recommendations** - -• Review the **addFooterDisplayFlags** function's conditional logic for setting `includeFlakyReportAllFooter`, specifically the conditions that check for flaky tests across all runs and in combination with previous results. - -• Verify all boolean comparisons and negations in the flaky test detection logic to ensure they are not inverted or contradictory. - -• Add explicit unit tests or debug traces to validate the flaky test count calculations when previous results are included to ensure accurate detection of flaky tests across runs. - -• Ensure the logic correctly distinguishes between three scenarios: (1) flaky tests exist across all runs with previous results, (2) no flaky tests exist across all runs with previous results, and (3) combined current and historical flaky tests, setting the flag appropriately for each case. ## AI Report diff --git a/images/ai-pr.png b/images/ai-pr.png deleted file mode 100644 index 897976bf..00000000 Binary files a/images/ai-pr.png and /dev/null differ diff --git a/images/ai.png b/images/ai.png deleted file mode 100644 index a2b8a991..00000000 Binary files a/images/ai.png and /dev/null differ diff --git a/images/all.png b/images/all.png deleted file mode 100644 index c71752f8..00000000 Binary files a/images/all.png and /dev/null differ diff --git a/images/annotations.png b/images/annotations.png deleted file mode 100644 index 0562c935..00000000 Binary files a/images/annotations.png and /dev/null differ diff --git a/images/custom-one.png b/images/custom-one.png deleted file mode 100644 index 36618722..00000000 Binary files a/images/custom-one.png and /dev/null differ diff --git a/images/failed-folded.png b/images/failed-folded.png deleted file mode 100644 index 0b4e198f..00000000 Binary files a/images/failed-folded.png and /dev/null differ diff --git a/images/failed-rate.png b/images/failed-rate.png deleted file mode 100644 index da3c0f05..00000000 Binary files a/images/failed-rate.png and /dev/null differ diff --git a/images/failed.png b/images/failed.png deleted file mode 100644 index 45e88831..00000000 Binary files a/images/failed.png and /dev/null differ diff --git a/images/flaky-rate.png b/images/flaky-rate.png deleted file mode 100644 index 95f02c6f..00000000 Binary files a/images/flaky-rate.png and /dev/null differ diff --git a/images/flaky.png b/images/flaky.png deleted file mode 100644 index 1e281646..00000000 Binary files a/images/flaky.png and /dev/null differ diff --git a/images/github-failed.png b/images/github-failed.png deleted file mode 100644 index 7cc22500..00000000 Binary files a/images/github-failed.png and /dev/null differ diff --git a/images/github-readme.png b/images/github-readme.png deleted file mode 100644 index 5c4b6db2..00000000 Binary files a/images/github-readme.png and /dev/null differ diff --git a/images/historical.png b/images/historical.png deleted file mode 100644 index 97abe02e..00000000 Binary files a/images/historical.png and /dev/null differ diff --git a/images/pr.png b/images/pr.png deleted file mode 100644 index 96fe5ff5..00000000 Binary files a/images/pr.png and /dev/null differ diff --git a/images/skipped.png b/images/skipped.png deleted file mode 100644 index e8f8056b..00000000 Binary files a/images/skipped.png and /dev/null differ diff --git a/images/suite-folded.png b/images/suite-folded.png deleted file mode 100644 index 39659dba..00000000 Binary files a/images/suite-folded.png and /dev/null differ diff --git a/images/suite-list.png b/images/suite-list.png deleted file mode 100644 index e4d7ea1c..00000000 Binary files a/images/suite-list.png and /dev/null differ diff --git a/images/summary.png b/images/summary.png deleted file mode 100644 index ffa8cb49..00000000 Binary files a/images/summary.png and /dev/null differ diff --git a/images/test-list.png b/images/test-list.png deleted file mode 100644 index da2e5297..00000000 Binary files a/images/test-list.png and /dev/null differ diff --git a/images/tests.png b/images/tests.png deleted file mode 100644 index 99997377..00000000 Binary files a/images/tests.png and /dev/null differ