Skip to content

Commit 9a319f6

Browse files
kalverrakalverra
committed
Merge branch 'main' of github.com:smartcontractkit/chainlink-testing-framework into skipFlakyTest
2 parents b4b7ee2 + 22779ab commit 9a319f6

File tree

12 files changed

+1438
-5
lines changed

12 files changed

+1438
-5
lines changed

‎tools/flakeguard/.flaky_test_db.json‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
{
2+
"github.com/smartcontractkit/chainlink/deployment/environment/crib::TestShouldProvideEnvironmentConfig": {
3+
"test_package": "github.com/smartcontractkit/chainlink/deployment/environment/crib",
4+
"test_name": "TestShouldProvideEnvironmentConfig",
5+
"jira_ticket": "DX-291",
6+
"skipped_at": "2025-03-24T16:13:41.618985Z"
7+
},
8+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_DifferentRmnNodesForDifferentChains": {
9+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
10+
"test_name": "TestRMN_DifferentRmnNodesForDifferentChains",
11+
"jira_ticket": "DX-202",
12+
"skipped_at": "2025-03-24T16:08:45.515288Z"
13+
},
14+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_GlobalCurseTwoMessagesOnTwoLanes": {
15+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
16+
"test_name": "TestRMN_GlobalCurseTwoMessagesOnTwoLanes",
17+
"jira_ticket": "DX-294",
18+
"skipped_at": "2025-03-24T16:08:45.515288Z"
19+
},
20+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_MultipleMessagesOnOneLaneNoWaitForExec": {
21+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
22+
"test_name": "TestRMN_MultipleMessagesOnOneLaneNoWaitForExec",
23+
"jira_ticket": "DX-201",
24+
"skipped_at": "2025-03-24T16:08:45.515288Z"
25+
},
26+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_NotEnoughObservers": {
27+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
28+
"test_name": "TestRMN_NotEnoughObservers",
29+
"jira_ticket": "DX-295",
30+
"skipped_at": "2025-03-24T16:05:20.79639Z"
31+
},
32+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_NotEnoughSigners": {
33+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
34+
"test_name": "TestRMN_NotEnoughSigners",
35+
"jira_ticket": "DX-296",
36+
"skipped_at": "2025-03-24T16:07:58.289626Z"
37+
},
38+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_TwoMessagesOnTwoLanesIncludingBatching": {
39+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
40+
"test_name": "TestRMN_TwoMessagesOnTwoLanesIncludingBatching",
41+
"jira_ticket": "DX-199",
42+
"skipped_at": "2025-03-24T16:07:10.367387Z"
43+
},
44+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_TwoMessagesOnTwoLanesIncludingBatchingWithTemporaryPause": {
45+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
46+
"test_name": "TestRMN_TwoMessagesOnTwoLanesIncludingBatchingWithTemporaryPause",
47+
"jira_ticket": "DX-123",
48+
"skipped_at": "2025-03-24T16:09:29.306211Z"
49+
},
50+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestRMN_TwoMessagesOneSourceChainCursed": {
51+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
52+
"test_name": "TestRMN_TwoMessagesOneSourceChainCursed",
53+
"jira_ticket": "DX-297",
54+
"skipped_at": "2025-03-24T16:13:24.263916Z"
55+
},
56+
"github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip::TestV1_5_Message_RMNRemote_Curse_Uncurse": {
57+
"test_package": "github.com/smartcontractkit/chainlink/integration-tests/smoke/ccip",
58+
"test_name": "TestV1_5_Message_RMNRemote_Curse_Uncurse",
59+
"jira_ticket": "DX-293",
60+
"skipped_at": "2025-03-24T16:06:00.975793Z"
61+
},
62+
"github.com/smartcontractkit/chainlink/v2::TestScripts/health/default": {
63+
"test_package": "github.com/smartcontractkit/chainlink/v2",
64+
"test_name": "TestScripts/health/default",
65+
"jira_ticket": "DX-109",
66+
"skipped_at": "2025-03-24T16:08:31.91215Z"
67+
}
68+
}

‎tools/flakeguard/.jira_assignee_mapping.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
[
2+
{
3+
"pattern": "github\\.com/smartcontractkit/chainlink/integration-tests/smoke/ccip.*",
4+
"assignee": "6175d3e016119e0069fdd14f"
5+
},
6+
{
7+
"pattern": "github\\.com/smartcontractkit/chainlink/integration-tests/smoke.*",
8+
"assignee": "638f597c3e79f12e57253913"
9+
},
10+
{
11+
"pattern": "github.com/smartcontractkit/chainlink/deployment/data-streams",
12+
"assignee": "641949fb1273131f2ae1dd4e"
13+
},
14+
{
15+
"pattern": "github.com/smartcontractkit/chainlink/system-tests/tests/smoke/capabilities.*",
16+
"assignee": "63beffc42a526608c5501530"
17+
},
18+
{
19+
"pattern": "github.com/smartcontractkit/chainlink/deployment.*",
20+
"assignee": "6175d3e016119e0069fdd14f"
21+
},
22+
{
23+
"pattern": ".*",
24+
"assignee": "638f597c3e79f12e57253913"
25+
}
26+
]

‎tools/flakeguard/.tool-versions‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
golang 1.24.0

‎tools/flakeguard/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,65 @@ Flakeguard offers two main commands:
2424

2525
- `find` identifies test packages affected by recent changes.
2626
- `run` executes tests multiple times to identify flaky tests
27+
- `create-tickets` automates the creation of Jira tickets for flaky tests detected by Flakeguard
2728

2829
Run with `--help` to see all flags for the commands.
2930

3031
### JSON Output
3132

3233
Both `find` and `run` commands support JSON output `--json`, making it easy to integrate Flakeguard with CI/CD pipelines and reporting tools.
3334

35+
### Creating JIRA Tickets
36+
The `create-tickets` command allows you to automate the creation of JIRA tickets for flaky tests. It reads test results from a CSV file (typically exported from a Splunk view) and creates tickets in JIRA.
37+
38+
```
39+
go run main.go create-tickets --jira-project=<JIRA_PROJECT_KEY> --flaky-test-json-db-path=<PATH_TO_FLAKY_TEST_DB_JSON> --assignee-mapping=<PATH_TO_JIRA_ASSIGNEE_MAPPING_JSON> --csv-path=<PATH_TO_CSV_FILE> [--skip-existing] [--dry-run]
40+
```
41+
42+
Example:
43+
```
44+
go run main.go create-tickets --jira-project=DX --flaky-test-json-db-path=.flaky_test_db.json --assignee-mapping=.jira_assignee_mapping.json --skip-existing --csv-path '1742825894_77903.csv'
45+
```
46+
47+
**Options:**
48+
49+
- `--jira-project`: The JIRA project key where tickets should be created (e.g., `DX`).
50+
- `--test-db-path`: The path to a JSON database (`.json`) that stores information about existing flaky test tickets.
51+
- `--assignee-mapping`: The path to a JSON file (`.json`) that maps test packages to JIRA assignees.
52+
- `--csv-path`: The path to the CSV file containing the flaky test results.
53+
- `--skip-existing`: (Optional) Skips creating tickets for tests that already have corresponding JIRA tickets in the database or JIRA.
54+
- `--dry-run`: (Optional) Performs a dry run without actually creating JIRA tickets.
55+
56+
**Environment Variables:**
57+
58+
- `JIRA_DOMAIN`: The domain of your JIRA instance.
59+
- `JIRA_EMAIL`: The email address used to authenticate with JIRA.
60+
- `JIRA_API_KEY`: The API key used to authenticate with JIRA.
61+
62+
### Managing Tickets
63+
The new tickets command lets you interactively manage your flaky test tickets stored in your local JSON database. With this command you can:
64+
65+
- Mark tests as skipped: Post a comment to the associated JIRA ticket and update the local DB.
66+
- Unskip tests: Remove the skipped status and post an unskip comment.
67+
- Navigate tickets: Use a TUI to browse through tickets.
68+
69+
```
70+
go run main.go tickets --test-db-path=.flaky_test_db.json --jira-comment
71+
```
72+
73+
**Options:**
74+
75+
- `--test-db-path`: The path to a JSON database (.json) that stores information about your flaky test tickets.
76+
- `--jira-comment`: If set to true, posts a comment to the corresponding JIRA ticket when marking a test as skipped or unskipped.
77+
`--dry-run`: (Optional) Runs the command in dry-run mode without posting comments to JIRA.
78+
- `--hide-skipped`: (Optional) If set, tickets already marked as skipped are hidden from the interface.
79+
80+
**Environment Variables:**
81+
82+
- `JIRA_DOMAIN`: The domain of your JIRA instance.
83+
- `JIRA_EMAIL`: The email address used to authenticate with JIRA.
84+
- `JIRA_API_KEY`: The API key used to authenticate with JIRA.
85+
3486
### Example Run
3587

3688
You can find example usage and see outputs with:

0 commit comments

Comments
 (0)