Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/product/test-analytics/img/TA-comment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/product/test-analytics/img/TA-dash.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
103 changes: 103 additions & 0 deletions docs/product/test-analytics/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
title: Test Analytics
sidebar_order: 135
description: Learn about how Sentry's Test Analytics tools can help you improve your code.
---

Sentry Test Analytics provides actionable insights into your CI test runs, helping you identify flaky tests, track failures, and optimize your test suite for faster, more reliable deployments.

## Why Use Test Analytics?

- **Spot Flaky Tests:** Quickly identify and address tests that fail intermittently, reducing noise and wasted debugging time
- **Track Failures and Trends:** Monitor cumulative failures, slowest tests, and overall test run efficiency to improve CI reliability
- **Actionable Insights:** See which lines of code failed within tests, confirm flaky tests, and prioritize fixes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is 'confirm flaky tests' different from 'spot flaky tests' above?

- **Integrated with Sentry:** Access test analytics alongside your error and performance data for a unified developer experience

## Getting Started

<Alert>
Sentry Test Analytics is currently in <strong>beta</strong>. Beta features are still a work in progress and may have bugs.
</Alert>

![Test Analytics Dashboard](./img/TA-dash.png)

To use Sentry Test Analytics, you’ll need to install the [Sentry app](https://github.com/apps/sentry-io) on your GitHub organization or specific repositories. Once installed, choose your JUnit XML Report framework and [set your permissions](#permissions-and-repository-tokens) using a repository secret.

<Alert>
**The only test result file format we support is JUnit XML**. Most test frameworks support outputting test results in this format, with some configuration.

Each testing framework will have slight differences in the JUnit XML format it outputs, but we aim to support all of them.
</Alert>

**There are two ways to upload your test results:**

### 1. Use GitHub Actions to Run Your CI
If you are already using GitHub Actions, you can use the `getsentry/prevent-action` script to upload your test results.

```yaml
- name: Upload test results to Sentry
if: ${{ !cancelled() }}
uses: getsentry/prevent-action
with:
token: ${{ secrets.SENTRY_PREVENT_TOKEN }}
```

You can also use [Open ID Connect](https://docs.github.com/en/actions/concepts/security/openid-connect) instead of the prevent tokenas a way to authenticate your GitHub Actions workflow.

### 2. Use [Sentry Prevent CLI](/product/test-analytics/sentry-prevent-cli/) to Upload Testing Reports
If you do not use Github Actions, install the Sentry Prevent CLI using `pip install sentry-prevent-cli` (for Python), or use a binary platform.

```bash {tabTitle:macos}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:linux}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:alpine linux}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:windows}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli.exe
```

```bash {tabTitle:linux arm64}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:alpine linux arm64}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

Then, upload your test results using the CLI:

```bash
sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN>
```

### Run Your Test Suite
Now that you've configured your CI to upload your test results, you can inspect the workflow logs to see if the call to Sentry succeeded. You need to have some failed tests to view the failed tests report.

## Viewing Test Analytics

After your workflow runs, view failed tests in the [failed tests dashboard](https://sentry.io/prevent/failed-tests/), or in your GitHub pull request comments.

![Test Analytics comment](./img/TA-comment.png)

You can see a complete overview of your test analytics in the [test analytics dashboard](sentry.io/prevent/).

## Permissions and Repository Tokens
When configuring Test Analytics, you'll be asked to choose your upload permissions. For GitHub Actions, you can use [OpenID Connect](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect), or generate a repository token. Using the CLI, generate a repository token.

You can find a list of repository tokens on the Sentry Prevent [tokens page](https://sentry.io/prevent/tokens/).

## Read Next

<PageGrid />
72 changes: 72 additions & 0 deletions docs/product/test-analytics/sentry-prevent-cli/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: Sentry Prevent CLI
sidebar_order: 10
description: Learn how to use the Sentry Prevent CLI to upload test results to Sentry.
---

## Installation

<Alert>
While the snippets below can be used to download and use the CLI directly, it is highly recommended to perform signature and SHASUM verification to ensure integrity of the CLI before use. See [Integrity Checking the Uploader](/product/test-analytics/sentry-prevent-cli/#integrity-checking-the-uploader) below for more information.
</Alert>

For `Python`, install the Sentry Prevent CLI using `pip install sentry-prevent-cli`. You need to have `Python3` as the default.

```bash
pip install sentry-prevent-cli
```

For `binary platforms`, use the following commands to download the latest version of the Sentry Prevent CLI.

```bash {tabTitle:macos}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:linux}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:alpine linux}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:windows}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli.exe
```

```bash {tabTitle:linux arm64}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```

```bash {tabTitle:alpine linux arm64}
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli
chmod +x sentry-prevent-cli
```
## Uploading Test Results

To upload test results, use the following command:

```bash
sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN>
```

## Integrity Checking the CLI

GitHub releases also includes the SHA for each asset. You can run

```bash
shasum -a 256 <filename>
```

or

```bash
sha256sum <filename>
```

and compare the checksum generated with that from GitHub releases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to have a screenshot or code snippet of what a successful execution would look like here, so the user knows if they got the right result?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that might be possible. I haven't had a chance to test it yet.


5 changes: 5 additions & 0 deletions redirects.js
Original file line number Diff line number Diff line change
Expand Up @@ -1213,6 +1213,11 @@ const userDocsRedirects = [
source: '/product/ai-in-sentry/seer/seer-privacy-and-security',
destination: '/product/ai-in-sentry/ai-privacy-and-security/',
},
// Redirects for Sentry Prevent AI
{
source: '/product/ai-in-sentry/sentry-prevent-ai/',
destination: '/product/ai-in-sentry/ai-code-review/',
},
// .NET profiling guide redirects - consolidate to main profiling page
{
source: '/platforms/dotnet/guides/uwp/profiling/:path*',
Expand Down
Loading