Skip to content

Conversation

angelosilvestre
Copy link
Contributor

@angelosilvestre angelosilvestre commented Jun 21, 2025

This PR adds a failing test, so we can verify the output. This is the full output:

00:01 +0: reports multiple failures
Golden scene has failures: Multiple Failures (✅ 0/2, ❌ 2/2, ❓ -1 / +1)

"❌ Red Rectangle" has a 51.19% (15000px) mismatch
"❌ A text" has an unexpected size (expected: Size(135.0, 68.0), actual: Size(160.0, 68.0))

Missing goldens:
❓ "Another Red Rectangle"

Extra (unexpected) candidates:
❓ "An unexpected Rectangle"

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Goldens failed with 4 mismatch(es)

When the exception was thrown, this was the stack:
#0      fail (package:matcher/src/expect/expect.dart:149:31)
#1      Gallery._compareGoldens (package:flutter_test_goldens/src/scenes/gallery.dart:565:7)
<asynchronous suspension>
#2      Gallery.renderOrCompareGolden (package:flutter_test_goldens/src/scenes/gallery.dart:324:7)
<asynchronous suspension>
#3      main.<anonymous closure> (file:///Users/angelosilvestre/dev/flutter_test_goldens/test_goldens/failure_reporting/failure_reporting_test.dart:9:5)
<asynchronous suspension>
#4      testGoldenScene.<anonymous closure> (package:flutter_test_goldens/src/test_runners.dart:261:9)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:193:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1064:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

The test description was:
  reports multiple failures
════════════════════════════════════════════════════════════════════════════════════════════════════
00:01 +0 -1: reports multiple failures [E]
  Test failed. See exception logs above.
  The test description was: reports multiple failures

@matthew-carroll
Copy link
Contributor

For the messaging, here's what I had in the presentation:

Golden scene failed (✅ 3/6, ❌ 3/6):
 ✅ Android caret
 ❌ Android collapsed handle (187px, 2.3%)
 ❌ Android expanded handles (234px, 4.8%)
 ✅ iOS caret
 ✅ iOS collapsed handle
 ❌ iOS expanded handles

First, can you adjust this test so that there's at least one success reported, too?

Second, can you try to make it look more like above, for the parts they have in common?

For the size mismatch, let's go with the following format:

 ❌ Android expanded handles (wrong size)
    - Golden size: (135, 68)
    - Candidate size: (160, 68)
    - Candidate is 25px too tall. Candidate has correct height.

@angelosilvestre
Copy link
Contributor Author

Updated:

Golden scene failed (✅ 1/3, ❌ 2/3, ❓ -1 / +1):
❌ Red Rectangle (15000px, 51.19%)
✅ A golden that passes
❌ A text (wrong size)
  - Golden size: (135.0, 68.0)
  - Candidate size: (160.0, 68.0)
  - Candidate is 25.0px too wide. Candidate has correct height.

Missing goldens:
❓ "Another Red Rectangle"

Extra (unexpected) candidates:
❓ "An unexpected Rectangle"

@angelosilvestre
Copy link
Contributor Author

Updated:

Golden scene failed (✅ 1/3, ❌ 2/3, ❓ -1 / +1):
❌ Android caret (15000px, 51.19%)
✅ Android drag handles
❌ Hint text (wrong size)
  - Golden size: (135, 68)
  - Candidate size: (160, 68)
  - Candidate is 25px too wide. Candidate has correct height.

Missing goldens:
❓ "iOS caret"

Extra (unexpected) candidates:
❓ "iOS drag handles"

Copy link
Contributor

@matthew-carroll matthew-carroll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - FYI, at least in this project, let's store the goldens next to the tests. I'm not sure there's any benefit to needing to expand another directory to see goldens generated by a given test file. But not gonna hold this up on that.

@matthew-carroll matthew-carroll marked this pull request as ready for review June 22, 2025 04:09
@matthew-carroll matthew-carroll merged commit c7e91b8 into main Jun 22, 2025
1 check passed
@matthew-carroll matthew-carroll deleted the add_failing_test branch June 22, 2025 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants