[Bug]: Test breakage detection tool does not handle tests consistently timing out and cancelled

[Abacn]

What happened?

It is found that

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml

has been timing out since Oct 30, but wasn't detected by the tool (it should reopen #34990)

relevant code path:

beam/.test-infra/tools/flaky_test_detection.py

Line 83 in 8ff2c94

github_workflow_failing_runs_url = f"https://github.com/{GIT_ORG}/beam/actions/{alert.workflow_filename}?query=is%3Afailure+branch%3Amaster"

EDIT:

actually the alert is fetched from grafana. We should change the alert criteria there

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner