Default failure summary for expanded reporter

[Telomelonia]

Summary

• Add failure_summary.dart with deterministic sorted output and (did not complete) annotations
• Call writeFailureSummary unconditionally in the expanded reporter on failure
• Update expanded reporter tests to expect the failure summary

[Telomelonia]

The issue core problem- not knowing which tests failed is solved... the line number is a nice-to-have, needs more work maybe extracting from trace

[jakemac53]

@Telomelonia I am curious specifically with the current output what is causing the output to be hard to read - is it that there are large stack traces/failure explanations for each failure?

In some sense I worry that if there are a lot of failures, this list at the bottom will now be hiding the actual reasons for failure, creating a different problem?

Maybe we should just print the first X failures at the bottom or something in the summary, followed by ... and X more or something like that.

[github-actions]

PR Health

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

[natebosch]

When running lots of tests, the failure details scroll off screen and you're left with just +152 -6: Some tests failed. with no easy way to see what actually broke.

This is not expected in typical cases when using the compact reporter. I'd understand the utility more clearly if we were adding this to the expanded reporter.

I'm not completely opposed to the idea of adding a summary, but we should first dig a bit more on the root cause. If there are common causes of excessively long output from the compact reporter I'd be more inclined to resolve those than to patch over the long output with even more output.

Can you clarify what type of output from successful tests is causing the failures to scroll away? Or is the problem with failure output that is too tall?

[Telomelonia]

@natebosch
You're right that the compact reporter shouldn't normally have this problem — the \r overwrite keeps things tight. But there are a few real-world cases where it breaks down:

• Tests that print() — forces newlines and breaks the \r overwrite. The framework has printOnFailure() but in practice a lot of tests just use print()
• Multiple failures with long stack traces / verbose matcher output stacking up
• Dependency warnings flooding output — like this person who got 63 lines of warnings per test run after upgrading Flutter, and their terminal couldn't even scroll back to failures anymore

That Flutter issue is actually titled "Flutter test doesn't report the failing tests in the summary" — same ask. And it's come up here before too (#263, #554).

I also agree with your point about the expanded reporter — this problem is worse there since every update is a new line. So I'd like to extend the summary to both compact and expanded rather than just compact.
The failures-only reporter helps but changes the whole output format — I thought --summary would be lighter, you keep your normal output and just get a quick list at the end when you opt in.

[Telomelonia]

Per @jakemac53's feedback I'll also cap it to the first 5 failures with "... and N more" so the summary itself doesn't get unwieldy.

[Telomelonia]

@natebosch cc: @jakemac53
I made the changes addressing the comments
I hope its much simpler and fixes problem
Please review

[Telomelonia]

@natebosch
Sorry for disturbing but can you allow ci to run, it must pass now
I am skipping the changelog for now

[kevmoo]

A changelog entry would be nice here!

[Telomelonia]

@kevmoo
I think ci will pass now 🤞