Skip to content

Commit b0068ca

Browse files
sfanahataShannon Anahatacoolguyzone
authored
New test analytics doc for docs.sentry (#14296)
## DESCRIBE YOUR PR Sentry Prevent is adding in Test Analytics to our product. - Rename `sentry-prevent-ai`: https://sentry-docs-git-sentry-prevent-test-analytics-docs.sentry.dev/product/ai-in-sentry/ai-code-review/ (should redirect when on https://sentry-docs-git-sentry-prevent-test-analytics-docs.sentry.dev/product/ai-in-sentry/sentry-prevent-ai/ ) - Test analytics page: https://sentry-docs-git-sentry-prevent-test-analytics-docs.sentry.dev/product/test-analytics/ - CLI: https://sentry-docs-git-sentry-prevent-test-analytics-docs.sentry.dev/product/test-analytics/sentry-prevent-cli/ ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. - [ ] Urgent deadline (GA date, etc.): <!-- ENTER DATE HERE --> - [x] Other deadline: Sept 23 - [ ] None: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it. Thanks in advance for your help! ## PRE-MERGE CHECKLIST *Make sure you've checked the following before merging your changes:* - [x] Checked Vercel preview for correctness, including links - [x] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [x] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) --------- Co-authored-by: Shannon Anahata <[email protected]> Co-authored-by: Alex Krawiec <[email protected]>
1 parent 4ed6118 commit b0068ca

File tree

4 files changed

+175
-0
lines changed

4 files changed

+175
-0
lines changed
130 KB
Loading
264 KB
Loading
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
title: Test Analytics
3+
sidebar_order: 135
4+
description: Learn how Sentry's Test Analytics tools can help you improve your code.
5+
---
6+
7+
Sentry Test Analytics provides actionable insights into your CI test runs, helping you identify flaky tests, track failures, and optimize your test suite for faster, more reliable deployments.
8+
9+
## Why Use Test Analytics?
10+
11+
- **Spot Flaky Tests:** Quickly identify and address tests that fail intermittently, reducing noise and wasted debugging time
12+
- **Track Failures and Trends:** Monitor cumulative failures, slowest tests, and overall test run efficiency to improve CI reliability
13+
- **Get Actionable Insights:** See which lines of code failed within tests, and prioritize fixes
14+
- **Leverage Sentry Integration:** Access test analytics alongside your error and performance data for a unified developer experience
15+
16+
## Getting Started
17+
18+
<Alert>
19+
Sentry Test Analytics is currently in <strong>beta</strong>. Beta features are still a work in progress and may have bugs.
20+
</Alert>
21+
22+
![Test Analytics Dashboard](./img/TA-dash.png)
23+
24+
To use Sentry Test Analytics, you’ll need to install the [Sentry app](https://github.com/apps/sentry-io) on your GitHub organization or specific repositories. Once installed, choose your JUnit XML Report framework and [set your permissions](#permissions-and-repository-tokens) using a repository secret.
25+
26+
<Alert>
27+
**The only test result file format we support is JUnit XML**. Most test frameworks support outputting test results in this format, with some configuration.
28+
29+
Each testing framework will have slight differences in the JUnit XML format it outputs, but we aim to support all of them.
30+
</Alert>
31+
32+
**There are two ways to upload your test results:**
33+
34+
### 1. Use GitHub Actions to Run Your CI
35+
If you are already using GitHub Actions, you can use the `getsentry/prevent-action` script to upload your test results.
36+
37+
```yaml
38+
- name: Upload test results to Sentry
39+
if: ${{ !cancelled() }}
40+
uses: getsentry/prevent-action
41+
with:
42+
token: ${{ secrets.SENTRY_PREVENT_TOKEN }}
43+
```
44+
45+
You can also use [Open ID Connect](https://docs.github.com/en/actions/concepts/security/openid-connect) instead of the prevent tokens a way to authenticate your GitHub Actions workflow.
46+
47+
### 2. Use [Sentry Prevent CLI](/product/test-analytics/sentry-prevent-cli/) to Upload Testing Reports
48+
If you do not use Github Actions, install the Sentry Prevent CLI using `pip install sentry-prevent-cli` (for Python), or use a binary platform.
49+
50+
```bash {tabTitle:macos}
51+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli
52+
chmod +x sentry-prevent-cli
53+
```
54+
55+
```bash {tabTitle:linux}
56+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli
57+
chmod +x sentry-prevent-cli
58+
```
59+
60+
```bash {tabTitle:alpine linux}
61+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli
62+
chmod +x sentry-prevent-cli
63+
```
64+
65+
```bash {tabTitle:windows}
66+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli.exe
67+
```
68+
69+
```bash {tabTitle:linux arm64}
70+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli
71+
chmod +x sentry-prevent-cli
72+
```
73+
74+
```bash {tabTitle:alpine linux arm64}
75+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli
76+
chmod +x sentry-prevent-cli
77+
```
78+
79+
Then, upload your test results using the CLI:
80+
81+
```bash
82+
sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN>
83+
```
84+
85+
### Run Your Test Suite
86+
Now that you've configured your CI to upload your test results, you can inspect the workflow logs to see if the call to Sentry succeeded. You need to have some failed tests to view the failed tests report.
87+
88+
## Viewing Test Analytics
89+
90+
After your workflow runs, view failed tests in the [failed tests dashboard](https://sentry.io/prevent/tests/?filterBy=failedTests), or in your GitHub pull request comments.
91+
92+
![Test Analytics comment](./img/TA-comment.png)
93+
94+
You can see a complete overview of your test analytics in the [test analytics dashboard](https://sentry.io/prevent/tests/).
95+
96+
## Permissions and Repository Tokens
97+
When configuring Test Analytics, you'll be asked to choose your upload permissions. For GitHub Actions, you can use [OpenID Connect](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect), or generate a repository token. Using the CLI, generate a repository token.
98+
99+
You can find a list of repository tokens on the Sentry Prevent [tokens page](https://sentry.io/prevent/tokens/).
100+
101+
## Read Next
102+
103+
<PageGrid />
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
title: Sentry Prevent CLI
3+
sidebar_order: 10
4+
description: Learn how to use the Sentry Prevent CLI to upload test results to Sentry.
5+
---
6+
7+
## Installation
8+
9+
<Alert>
10+
While the snippets below can be used to download and use the CLI directly, it is highly recommended to perform signature and SHASUM verification to ensure integrity of the CLI before use. See [Integrity Checking the Uploader](/product/test-analytics/sentry-prevent-cli/#integrity-checking-the-cli) below for more information.
11+
</Alert>
12+
13+
For `Python`, install the Sentry Prevent CLI using `pip install sentry-prevent-cli`. You need to have `Python3` as the default.
14+
15+
```bash
16+
pip install sentry-prevent-cli
17+
```
18+
19+
For `binary platforms`, use the following commands to download the latest version of the Sentry Prevent CLI.
20+
21+
```bash {tabTitle:macos}
22+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli_macos
23+
chmod +x sentry-prevent-cli_macos
24+
```
25+
26+
```bash {tabTitle:linux x64_64}
27+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli_linux_x64_64
28+
chmod +x sentry-prevent-cli_linux_x64_64
29+
```
30+
31+
```bash {tabTitle:linux arm64}
32+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli_linux_arm64
33+
chmod +x sentry-prevent-cli_linux_arm64
34+
```
35+
36+
```bash {tabTitle:alpine linux x64_64}
37+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli_alpine_x64_64
38+
chmod +x sentry-prevent-cli_alpine_x64_64
39+
```
40+
41+
```bash {tabTitle:alpine linux arm64}
42+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli_alpine_arm64
43+
chmod +x sentry-prevent-cli_alpine_arm64
44+
```
45+
46+
```bash {tabTitle:windows}
47+
curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli_windows.exe
48+
```
49+
## Uploading Test Results
50+
51+
To upload test results, use the following command:
52+
53+
```bash
54+
sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN>
55+
```
56+
57+
## Integrity Checking the CLI
58+
59+
GitHub releases also includes the SHA for each asset. You can run
60+
61+
```bash
62+
shasum -a 256 <filename>
63+
```
64+
65+
or
66+
67+
```bash
68+
sha256sum <filename>
69+
```
70+
71+
and compare the checksum generated with that from GitHub releases.
72+

0 commit comments

Comments
 (0)