|
| 1 | +# Flakeguard |
| 2 | + |
| 3 | +**Flakeguard** is a tool designed to help identify flaky tests within a Go project. Flaky tests are tests that intermittently fail without changes to the code, often due to race conditions or other non-deterministic behavior. Flakeguard assists by analyzing the impact of code changes on test packages and by running tests multiple times to determine stability. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- **Identify Impacted Tests**: Detects test packages that may be affected by changes in your Go project files. |
| 8 | +- **Run Tests for Flakiness**: Runs tests multiple times to determine their flakiness. |
| 9 | +- **Output Results in JSON**: Allows easy integration with CI pipelines and custom reporting. |
| 10 | +- **Supports Exclusion Lists**: Configurable to exclude specified packages or paths from the analysis. |
| 11 | +- **Recursive Dependency Analysis**: Detects all impacted packages through dependency levels. |
| 12 | + |
| 13 | +## Installation |
| 14 | + |
| 15 | +To install `flakeguard` CLI, you need to have Go installed on your machine. With Go installed, run the following command: |
| 16 | + |
| 17 | +```sh |
| 18 | +go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@latest |
| 19 | +``` |
| 20 | + |
| 21 | +## Usage |
| 22 | + |
| 23 | +Flakeguard offers two main commands: `find` and `run`. |
| 24 | + |
| 25 | +### `find` Command |
| 26 | + |
| 27 | +The `find` command identifies test packages affected by recent changes. |
| 28 | + |
| 29 | +#### Flags for `find` Command |
| 30 | + |
| 31 | +- `--project-path, -r` : Path to the Go project. Defaults to the current directory. |
| 32 | +- `--base-ref` (required): Git reference (branch, tag, or commit) for comparing changes. |
| 33 | +- `--verbose, -v` : Enables verbose mode. |
| 34 | +- `--json` : Outputs results in JSON format. |
| 35 | +- `--filter-empty-tests` : Filters out test packages with no actual test functions (can be slow on large projects). |
| 36 | +- `--excludes` : List of paths to exclude. |
| 37 | +- `--levels, -l` : Levels of recursion for dependency search, with `0` for unlimited. `2` by default. |
| 38 | +- `--find-by-test-files-diff` : Enable mode to find affected test packages by changes in test files. |
| 39 | +- `--find-by-affected-packages` : Enable mode to find affected test packages based on changes in any project package. |
| 40 | +- `--only-show-changed-test-files` : Display only changed test files and exit. |
| 41 | + |
| 42 | +### `run` Command |
| 43 | + |
| 44 | +The `run` command executes tests multiple times to identify flaky tests. |
| 45 | + |
| 46 | +#### Flags for `run` Command |
| 47 | + |
| 48 | +- `--project-path, -r` : Path to the Go project. Defaults to the current directory. |
| 49 | +- `--test-packages-json` : JSON-encoded string of test packages to run. |
| 50 | +- `--test-packages` : List of test packages to run. |
| 51 | +- `--run-count, -c` : Number of times to run the tests. |
| 52 | +- `--race` : Enable race condition detection. |
| 53 | +- `--output-json` : File path to output test results in JSON format. |
| 54 | +- `--threshold` : Threshold (0-1) for determining flakiness (e.g., `0.8` to pass if 80% successful). |
| 55 | +- `--skip-tests` : List of tests to skip. |
| 56 | +- `--fail-fast` : Stop on the first failure if threshold is set to `1.0`. |
| 57 | + |
| 58 | +### JSON Output |
| 59 | + |
| 60 | +Both `find` and `run` commands support JSON output `--json`, making it easy to integrate Flakeguard with CI/CD pipelines and reporting tools. |
0 commit comments