TT-1995 Prevent double-counting of test failures in Flakeguard

[lukaszcl]

This PR introduces a custom Go Test JSON transformer and integrates it into Flakeguard to enhance test failure reporting.

Key Changes:

1. New Go Test JSON Transformer - Transforms go test -json output to ensure that parent tests are not marked as failures when only their subtests fail (i.e., no direct parent failure).

2. Flakeguard Integration - The transformer is integrated into Flakeguard to fix TT-1995, preventing the double-counting of test failures.

3. Flakeguard Summary Report Updates - Added improvements to the Flakeguard Summary report.

The transformer is now tested with real data using the Flakeguard Nightly Workflow in core repo, ensuring accuracy and reliability in real scenarios.

To Do:

• Test the PR with Flakeguard Nightly Workflow in core repo
  • https://github.com/smartcontractkit/chainlink/actions/runs/13567803898
  • https://github.com/smartcontractkit/chainlink/actions/runs/13568685948
  • https://github.com/smartcontractkit/chainlink/actions/runs/13568686834
  • https://github.com/smartcontractkit/chainlink/actions/runs/13571219748
  • https://github.com/smartcontractkit/chainlink/actions/runs/13571218966
  • https://github.com/smartcontractkit/chainlink/actions/runs/13572150487
  • https://github.com/smartcontractkit/chainlink/actions/runs/13584595538

---

Below is a summarization created by an LLM (gpt-4-0125-preview). Be mindful of hallucinations and verify accuracy.

Why

The changes introduce a new feature to ignore parent test failures when only their subtests fail, enhancing the test result analysis process. This is particularly useful in large test suites where failing subtests may not necessarily indicate a problem with the parent test logic itself. Additionally, the presentation of flaky and panicked tests has been improved in the summary table for better clarity.

What

• tools/flakeguard/cmd/run.go: Added a new flag ignore-parent-failures-on-subtests to enable the feature of ignoring parent test failures when only subtests fail.
• tools/flakeguard/go-test-transform/README.md: Created a README file for the go-test-transform utility, explaining its purpose, features, and usage.
• tools/flakeguard/go-test-transform/cmd/main.go: Implemented the main command for the go-test-transform utility, including command-line flags for input/output options and the ignore-all mechanism.
• tools/flakeguard/go-test-transform/pkg/transformer/options.go: Defined options for the test transformer, specifically the option to ignore all subtest failures.
• tools/flakeguard/go-test-transform/pkg/transformer/transformer.go: Implemented the logic for transforming go test JSON output to handle subtest failures more intelligently.
• tools/flakeguard/go-test-transform/pkg/transformer/transformer_test.go: Added comprehensive tests to verify the correct behavior of the transformer logic under various scenarios, including handling special characters in test names and dealing with fuzz tests.
• tools/flakeguard/reports/presentation.go: Modified the summary table rendering to include rows for unique flaky tests and unique panicked tests, improving the presentation of test results.
• tools/flakeguard/runner/runner.go: Integrated the go-test-transform utility's transformation logic into the test runner to apply the ignore parent failures feature and handle transformed output files.

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
31.0% Duplication on New Code (required ≤ 5%)
1 New Critical Issues (required ≤ 0)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube IDE