convenient_test_manager_dart runs forever when an assertion error occurs

[lehmannk]

I have a very simple Test which should produce an assertion error because "DOES NOT EXIST" does not exist in the widget tree:

    group('simple test group', () {
      tTestWidgets('simple test', (t) async {
        await t.get(find.text('DOES NOT EXIST')).should(findsOneWidget);
      });
    });
  });

Now, while the Manager UI (convenient_test_manager) properly detects the missing widget and prints the assertion failure the CLI version for CI convenient_test_manager_dart runs forever and never terminates. Here is the log:

2023-10-30T14:28:48.447752Z|debug|ReportHandlerService|handleReportLogEntry called
2023-10-30T14:28:48.572984Z|debug|ReportHandlerService|Snapshot
2023-10-30T14:28:48.775753Z|debug|ReportHandlerService|StateChange: testName=simple test group simple test state=status: complete
result: error
2023-10-30T14:28:48.791420Z|debug|ReportHandlerService|Error: 'package:flutter_test/src/binding.dart': Failed assertion: line 950 pos 14: '_pendingExceptionDetails != null': A test overrode FlutterError.onError but either failed to return it to its original state, or had unexpected additional errors that it could not handle. Typically, this is caused by using expect() before restoring FlutterError.onError. stack=dart:core-patch/errors_patch.dart 51:61       _AssertionError._doThrowNew
dart:core-patch/errors_patch.dart 40:5        _AssertionError._throwNew
package:flutter_test/src/binding.dart 950:14  TestWidgetsFlutterBinding._runTest.handleUncaughtError
package:flutter_test/src/binding.dart 955:9   TestWidgetsFlutterBinding._runTest.<fn>
dart:async/zone.dart 1081:14                  _Zone._processUncaughtError
dart:async/zone.dart 1285:5                   _CustomZone.handleUncaughtError
dart:async/future_impl.dart 790:16            Future._propagateToListeners
dart:async/future_impl.dart 649:5             Future._completeError
dart:async/zone.dart 1423:47                  _rootRunBinary
dart:async/zone.dart 1315:19                  _CustomZone.runBinary
dart:async/future_impl.dart 171:22            _FutureListener.handleError
dart:async/future_impl.dart 852:47            Future._propagateToListeners.handleError
dart:async/future_impl.dart 873:13            Future._propagateToListeners
dart:async/future_impl.dart 649:5             Future._completeError
dart:async/zone.dart 1407:47                  _rootRunUnary
dart:async/zone.dart 1308:19                  _CustomZone.runUnary
dart:async/zone.dart 1399:13                  _rootRun
dart:async/zone.dart 1301:19                  _CustomZone.run
dart:async/zone.dart 1209:7                   _CustomZone.runGuarded
dart:async/zone.dart 1249:23                  _CustomZone.bindCallbackGuarded.<fn>
dart:async/schedule_microtask.dart 40:21      _microtaskLoop
dart:async/schedule_microtask.dart 49:5       _startMicrotaskLoop
2023-10-30T14:28:48.791420Z|debug|ReportHandlerService|handleReportLogEntry called
🚀 [STATUS] [0:16] [1x completeFailureOrError] [Test=simple test group simple test]
🚀 [STATUS] [0:18] [1x completeFailureOrError] [Test=simple test group simple test]
2023-10-30T14:28:53.244813Z|info|main|monitorWorkerAvailable check
🚀 [STATUS] [0:20] [1x completeFailureOrError] [Test=simple test group simple test]
🚀 [STATUS] [0:22] [1x completeFailureOrError] [Test=simple test group simple test]
🚀 [STATUS] [0:24] [1x completeFailureOrError] [Test=simple test group simple test]
2023-10-30T14:28:58.260562Z|info|main|monitorWorkerAvailable check

... [running forever from here]

What am I doing wrong? AND: observing the logs in a CI build, what is the best trigger to detect a successful testrun or a failed testrun? The manager logs are full of debug messages etc. It is very difficult to actually identify what went wrong with regards to test assertions. How can I turn debug logging off for example?

[fzyzcjy]

That looks like a bug! If you only have one test, 🚀 [STATUS] [0:24] [1x completeFailureOrError] [Test=simple test group simple test] means that, it successfully knows there is one failed (errored) test. Looks like the run status

flutter_convenient_test/packages/convenient_test_manager_dart/bin/convenient_test_manager_dart.dart

Line 102 in c24191d

Future<void> _awaitSuperRunStatusTestAllDone() async {

is somehow not updated.

observing the logs in a CI build, what is the best trigger to detect a successful testrun or a failed testrun?

By nonzero exit code, it is a failed run

The manager logs are full of debug messages etc. It is very difficult to actually identify what went wrong with regards to test assertions. How can I turn debug logging off for example?

The option to control it is in https://github.com/fzyzcjy/flutter_convenient_test/blob/c24191d1298b1f969b2053b8b98891d58161cf15/packages/convenient_test_common_dart/lib/src/common_dart/front_log.dart#L16C10-L16C10. Feel free to PR to make it configurable from outside! (Or hack the code and modify it directly)

[lehmannk]

I think I found the solution, it is a general problem with integration_test and overriding FlutterError.onError the solution is given in flutter/flutter#34499 (comment). I have implemented the fix in convenient_test_dev locally and now exceptions are properly handled. Also, the CLI manager terminates properly in case of errors. I will prepare a PR for this tomorrow ;-)

[fzyzcjy]

Great job!

[lehmannk]

I cannot push my bugfix branch bugfix/prevent_onError_override_exceptions, do you need to set any permissions for me?

Push failed remote: Permission to fzyzcjy/flutter_convenient_test.git denied to lehmannk. unable to access 'https://github.com/fzyzcjy/flutter_convenient_test.git/': The requested URL returned error: 403

[fzyzcjy]

IMHO a standard workflow in GitHub involves things like "pull request" and "fork", https://github.com/features/code-review

[lehmannk]

oh sorry, I wasn't aware I need to fork first. Here is my PR: #351

[lehmannk]

Ok, for everyone interested. Here is the solution to avoid the onError problem and endless headless runs. Simply backup & restore the onErrorhandler in your ConvenientTestSlot override:

 @override
  Future<void> appMain(AppMainExecuteMode mode) async {
    // onError overrides need to be restored in order to let the headless test manager terminate properly in case of exceptions
    // see https://github.com/fzyzcjy/flutter_convenient_test/pull/351
    FlutterExceptionHandler? originalOnError = FlutterError.onError;
    await app.main();
    FlutterError.onError = originalOnError;
  }

[fzyzcjy]

Good job! 🎉