-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New test analytics doc for docs.sentry #14296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
09d73f4
new test analytics doc for sentry
2cf38dd
adding more details, screenshots, and URLs
6988e7f
adding image
d8b83cc
adding proper links
52d625b
updating test doc and adding a CLI doc
9703fbd
refreshing preview
acf0fb0
up
6248533
Merge master into sentry-prevent-test-analytics-docs
0c9e6f5
Update AI Code Review docs with master changes
7e9bd4d
updating proper bash syntax and SHA instructions
596e209
Merge master to get AI Code Review changes from PR #14991
e7ae04c
edits to the TA doc
6fcea38
fixing broken link
5b060af
Remove outdated sentry-prevent-review.png image to match master
da21e11
Remove duplicate redirect to match master - keep only the wildcard ve…
a02bd8b
updating binary commands
18a4a32
Update docs/product/test-analytics/index.mdx
sfanahata 76802ef
Update docs/product/test-analytics/index.mdx
sfanahata 32df2b7
Update docs/product/test-analytics/index.mdx
sfanahata c499e57
Update docs/product/test-analytics/index.mdx
sfanahata 68b75c9
Update docs/product/test-analytics/index.mdx
sfanahata 5e4b806
fixing broken links
0f82bae
trying to force refresh the build preview
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| --- | ||
| title: Test Analytics | ||
| sidebar_order: 135 | ||
| description: Learn how Sentry's Test Analytics tools can help you improve your code. | ||
| --- | ||
|
|
||
| Sentry Test Analytics provides actionable insights into your CI test runs, helping you identify flaky tests, track failures, and optimize your test suite for faster, more reliable deployments. | ||
|
|
||
| ## Why Use Test Analytics? | ||
|
|
||
| - **Spot Flaky Tests:** Quickly identify and address tests that fail intermittently, reducing noise and wasted debugging time | ||
| - **Track Failures and Trends:** Monitor cumulative failures, slowest tests, and overall test run efficiency to improve CI reliability | ||
| - **Get Actionable Insights:** See which lines of code failed within tests, and prioritize fixes | ||
| - **Leverage Sentry Integration:** Access test analytics alongside your error and performance data for a unified developer experience | ||
|
|
||
| ## Getting Started | ||
|
|
||
| <Alert> | ||
| Sentry Test Analytics is currently in <strong>beta</strong>. Beta features are still a work in progress and may have bugs. | ||
| </Alert> | ||
|
|
||
|  | ||
|
|
||
| To use Sentry Test Analytics, you’ll need to install the [Sentry app](https://github.com/apps/sentry-io) on your GitHub organization or specific repositories. Once installed, choose your JUnit XML Report framework and [set your permissions](#permissions-and-repository-tokens) using a repository secret. | ||
|
|
||
| <Alert> | ||
| **The only test result file format we support is JUnit XML**. Most test frameworks support outputting test results in this format, with some configuration. | ||
|
|
||
| Each testing framework will have slight differences in the JUnit XML format it outputs, but we aim to support all of them. | ||
| </Alert> | ||
|
|
||
| **There are two ways to upload your test results:** | ||
|
|
||
| ### 1. Use GitHub Actions to Run Your CI | ||
| If you are already using GitHub Actions, you can use the `getsentry/prevent-action` script to upload your test results. | ||
|
|
||
| ```yaml | ||
| - name: Upload test results to Sentry | ||
| if: ${{ !cancelled() }} | ||
| uses: getsentry/prevent-action | ||
| with: | ||
| token: ${{ secrets.SENTRY_PREVENT_TOKEN }} | ||
| ``` | ||
|
|
||
| You can also use [Open ID Connect](https://docs.github.com/en/actions/concepts/security/openid-connect) instead of the prevent tokens a way to authenticate your GitHub Actions workflow. | ||
|
|
||
| ### 2. Use [Sentry Prevent CLI](/product/test-analytics/sentry-prevent-cli/) to Upload Testing Reports | ||
| If you do not use Github Actions, install the Sentry Prevent CLI using `pip install sentry-prevent-cli` (for Python), or use a binary platform. | ||
|
|
||
| ```bash {tabTitle:macos} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli | ||
| chmod +x sentry-prevent-cli | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:linux} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli | ||
| chmod +x sentry-prevent-cli | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:alpine linux} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli | ||
| chmod +x sentry-prevent-cli | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:windows} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli.exe | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:linux arm64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli | ||
| chmod +x sentry-prevent-cli | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:alpine linux arm64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli | ||
| chmod +x sentry-prevent-cli | ||
| ``` | ||
|
|
||
| Then, upload your test results using the CLI: | ||
|
|
||
| ```bash | ||
| sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN> | ||
| ``` | ||
|
|
||
| ### Run Your Test Suite | ||
| Now that you've configured your CI to upload your test results, you can inspect the workflow logs to see if the call to Sentry succeeded. You need to have some failed tests to view the failed tests report. | ||
|
|
||
| ## Viewing Test Analytics | ||
|
|
||
| After your workflow runs, view failed tests in the [failed tests dashboard](https://sentry.io/prevent/tests/?filterBy=failedTests), or in your GitHub pull request comments. | ||
|
|
||
|  | ||
|
|
||
| You can see a complete overview of your test analytics in the [test analytics dashboard](https://sentry.io/prevent/tests/). | ||
|
|
||
| ## Permissions and Repository Tokens | ||
| When configuring Test Analytics, you'll be asked to choose your upload permissions. For GitHub Actions, you can use [OpenID Connect](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect), or generate a repository token. Using the CLI, generate a repository token. | ||
|
|
||
| You can find a list of repository tokens on the Sentry Prevent [tokens page](https://sentry.io/prevent/tokens/). | ||
|
|
||
| ## Read Next | ||
|
|
||
| <PageGrid /> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| title: Sentry Prevent CLI | ||
| sidebar_order: 10 | ||
| description: Learn how to use the Sentry Prevent CLI to upload test results to Sentry. | ||
| --- | ||
|
|
||
| ## Installation | ||
|
|
||
| <Alert> | ||
| While the snippets below can be used to download and use the CLI directly, it is highly recommended to perform signature and SHASUM verification to ensure integrity of the CLI before use. See [Integrity Checking the Uploader](/product/test-analytics/sentry-prevent-cli/#integrity-checking-the-cli) below for more information. | ||
| </Alert> | ||
|
|
||
| For `Python`, install the Sentry Prevent CLI using `pip install sentry-prevent-cli`. You need to have `Python3` as the default. | ||
|
|
||
| ```bash | ||
| pip install sentry-prevent-cli | ||
| ``` | ||
|
|
||
| For `binary platforms`, use the following commands to download the latest version of the Sentry Prevent CLI. | ||
|
|
||
| ```bash {tabTitle:macos} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-macos -o sentry-prevent-cli_macos | ||
| chmod +x sentry-prevent-cli_macos | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:linux x64_64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux -o sentry-prevent-cli_linux_x64_64 | ||
| chmod +x sentry-prevent-cli_linux_x64_64 | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:linux arm64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-linux-arm64 -o sentry-prevent-cli_linux_arm64 | ||
| chmod +x sentry-prevent-cli_linux_arm64 | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:alpine linux x64_64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine -o sentry-prevent-cli_alpine_x64_64 | ||
| chmod +x sentry-prevent-cli_alpine_x64_64 | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:alpine linux arm64} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-alpine-arm64 -o sentry-prevent-cli_alpine_arm64 | ||
| chmod +x sentry-prevent-cli_alpine_arm64 | ||
| ``` | ||
|
|
||
| ```bash {tabTitle:windows} | ||
| curl -L https://github.com/getsentry/sentry-prevent-cli/releases/latest/download/sentry-prevent-cli-windows.exe -o sentry-prevent-cli_windows.exe | ||
| ``` | ||
| ## Uploading Test Results | ||
|
|
||
| To upload test results, use the following command: | ||
|
|
||
| ```bash | ||
| sentry-prevent-cli upload --report-type test-results --token <PREVENT_TOKEN> | ||
| ``` | ||
|
|
||
| ## Integrity Checking the CLI | ||
|
|
||
| GitHub releases also includes the SHA for each asset. You can run | ||
|
|
||
| ```bash | ||
| shasum -a 256 <filename> | ||
| ``` | ||
|
|
||
| or | ||
|
|
||
| ```bash | ||
| sha256sum <filename> | ||
| ``` | ||
|
|
||
| and compare the checksum generated with that from GitHub releases. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to have a screenshot or code snippet of what a successful execution would look like here, so the user knows if they got the right result?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that might be possible. I haven't had a chance to test it yet.