ci: Add automatic flaky test detector

[nicohrubec]

Manually checking for flakes and opening issues is a bit annoying. I was thinking we could add a ci workflow to automate this. The action only runs when merging to develop. Could also be done on PRs but seems unnecessarily complicated. My thinking is that for a push to develop to happen, all the test must first have passed in the original PR. Therefore if the test then fails on develop we know it's a flake. Open for ideas/improvements/cleanups or let me know if there might be any cases I am missing that could lead to false positives.

Example issue created with this: #18693

It doesn't get all the details but I think basically the most important is a link to the run so we can then investigate further. Also the logic for creating the issues is a bit ugly, but not sure if we can make it cleaner given that I want to create one issue per failed test not dump it all into one issue.

[github-actions]

node-overhead report 🧳

Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.

Scenario	Requests/s	% of Baseline	Prev. Requests/s	Change %
GET Baseline	8,988	-	8,586	+5%
GET With Sentry	1,736	19%	1,602	+8%
GET With Sentry (error only)	6,022	67%	5,921	+2%
POST Baseline	1,196	-	1,174	+2%
POST With Sentry	610	51%	572	+7%
POST With Sentry (error only)	1,063	89%	1,015	+5%
MYSQL Baseline	3,383	-	3,167	+7%
MYSQL With Sentry	522	15%	405	+29%
MYSQL With Sentry (error only)	2,732	81%	2,609	+5%

View base workflow run

[cursor]

String replace special patterns corrupt job name substitution

The replace() calls interpret special dollar-sign sequences in the replacement string ($&, $', $$, $1, etc.) rather than treating them literally. If a job name happens to contain patterns like $& or $1, the issue title and body would contain the matched template placeholder or captured groups instead of the literal job name. While job names rarely contain such patterns, this could cause confusing issue titles when they do.

 

[JPeer264]

I'm not 100% sure on this. The new ticket is unassigned by default and, I think, it doesn't get an SLA set in Linear.

Manually checking for flakes and opening issues is a bit annoying

To understand this use case - don't we get reported within Sentry if there are flaky tests? We could simply adapt the alert: https://sentry.sentry.io/issues/alerts/rules/details/267675/ (which we can also assign to people)

[nicohrubec]

@JPeer264 We only get alerted if there are more than 10 flaky tests detected within an hour, which is a good start but I think not ideal (quite arbitrary since it depends on how much people are pushing to PRs, for instance).

My main goal was to increase visibility of flaky tests, which I think we already get from having them in the issue feed instead of needing to manually check for failing tests and then creating issues so somebody can fix them. I am not exactly sure how/when SLAs are assigned , but we can probably get an SLA by assigning the correct label? Automatically having someone assigned sounds difficult to me though, maybe flaky CI issues can just be another stream that we can look out for during triage.

Happy to explore other options as well (e.g. using the alerts we have), just wanted to put this out to see if we can maybe improve the process around this a bit