Integration Tests ALWAYS Pass Locally But Fail In CodeMagic

[danielccdk]

If you are looking for help, please confirm the following...

• I confirm I have searched the Docs, Codemagic Sample Projects, and GitHub Discussions.
• I confirm the software versions on my local computer match the ones in CI environment (e.g. Xcode version, Gradle version). See here
• I confirm I am able to do a fresh clone of the repository and build the project on my local computer.

Which mobile framework are you using?

Flutter (Dart)

Steps to reproduce

The codemagic.yaml script:

flutter emulators --launch apple_ios_simulator

      # Wait a few seconds for the emulator to fully launch
      sleep 10

      # Run the integration test using flutter drive and define the testing environment variable
      flutter drive \
        --driver=integration_test/driver/driver.dart \
        --target=integration_test/test_master.dart \
        --dart-define=ENV=testing

Expected results

All integration tests (~35) pass locally.
The most basic of tests are failing in codemagic..

Actual results

I commented out all of the tests except one.
I get..

flutter: [ widget.manheimEnabled ]  true
flutter: Appraisal or it's VIN is null
flutter: initializeBlackbook() ERROR while loading blackbook data: type 'Null' is not a subtype of type 'BlackbookModel'
flutter: getManheimYears()
flutter: [ INSIGHTS API REQUEST HEADERS ]: {Content-type: application/json, Accept: application/json, x-rli-version: 3.0.4, x-rli-environment: ios-diy, context-id: 3693, context-account: 111118, context-app-origin: mobileViewAppraisal}
flutter: [ INSIGHTS API REQUEST HEADERS ]: {Content-type: application/json, Accept: application/json, x-rli-version: 3.0.4, x-rli-environment: ios-diy, context-id: 3693, context-account: 111118, context-app-origin: mobileViewAppraisal}
flutter: [ UNmounting appraisal insights ]
flutter: [ Attempt to refresh token ]
flutter: getManheimMakes(2017)
flutter: [ handleUnauthorizedRequest() ERROR in api.dart ]
flutter: [ bugsnag notifyError() error ]: Exception: You must start or attach bugsnag before calling any other methods
flutter: [ sendHttpRequest() ERROR from api.dart ] HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: [ getBlackbookMakes() ERROR ]: HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: [ bugsnag notifyError() error ]: Exception: You must start or attach bugsnag before calling any other methods
flutter: HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: #0      Api.sendHttpRequest (package:rli_mobile/api.dart:2290:9)
flutter: <asynchronous suspension>
flutter: #1      Api.getBlackbookMakes (package:rli_mobile/api.dart:1404:11)
flutter: <asynchronous suspension>
flutter: #2      BlackbookProvider.getMakes (package:rli_mobile/common/blackbook/blackbook_provider.dart:82:15)
flutter: <asynchronous suspension>
flutter: #3      _BlackbookCardState._initBlackBookCardData.<anonymous closure> (package:rli_mobile/common/blackbook/blackbook.dart:166:63)
flutter: <asynchronous suspension>
flutter: #4      _BlackbookCardState._initBlackBookCardData (package:rli_mobile/common/blackbook/blackbook.dart:166:11)
flutter: <asynchronous suspension>
flutter: #5      _FutureBuilderState._subscribe.<anonymous closure> (package:flutter/src/widgets/async.dart:638:31)
flutter: <asynchronous suspension>
flutter:
flutter: [ Attempt to refresh token ]
flutter: [ handleUnauthorizedRequest() ERROR in api.dart ]
flutter: [ bugsnag notifyError() error ]: Exception: You must start or attach bugsnag before calling any other methods
flutter: [ sendHttpRequest() ERROR from api.dart ] HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: [ getManheimMakes() ERROR ]: HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: [ bugsnag notifyError() error ]: Exception: You must start or attach bugsnag before calling any other methods
flutter: ERROR while loading manheim makes: HTTP request failed, status code: 401, body: {"error":"Authorization failed.","message":"We were unable to verify this user."}
flutter: [ bugsnag notifyError() error ]: Exception: You must start or attach bugsnag before calling any other methods
flutter: initialize() ERROR while loading manheim data: A ManheimProvider was used after being disposed.
flutter: Once you have called dispose() on a ManheimProvider, it can no longer be used.
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure was thrown running a test:
flutter: Expected: exactly one matching candidate
flutter:   Actual: _TextWidgetFinder:<Found 0 widgets with text "2017 Jeep Wrangler": []>
flutter:    Which: means none were found but one was expected
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4      main.<anonymous closure>.<anonymous closure> (file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart:66:7)
flutter: <asynchronous suspension>
flutter: #5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
flutter: <asynchronous suspension>
flutter: #6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
flutter: <asynchronous suspension>
flutter: #7      TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:824:12)
flutter: <asynchronous suspension>
flutter:
flutter: This was caught by the test expectation on the following line:
flutter:   file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart line 66
flutter: The test description was:
flutter:   New appraisal is instantly included in appraisals list view
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 01:24 +1: add appraisal test New appraisal is instantly included in appraisals list view [E]
flutter:   Test failed. See exception logs above.

The test description was: New appraisal is instantly included in appraisals list view
flutter:

The API failure works everytime locally, also.
I'm not sure if the UI is sticking because the API fails in codemagic?
(Is there something additional I need to do in order to get that API to work in a codemagic env?)

Additionally I have HAD issues with the permission handler getting dismissed in the past but.. even if it doesn't get dismissed, all tests still pass locally. I'm a bit at a loss currently.

Thank you so much!!

Build id (optional)

67363455833f3c00c0e8b7a2

[danielccdk]

How do we set a specific simulator/iOS for flutter integration tests or driver? Even the example from codemagic docs doesn't work.

scripts:
  - name: Emulator tests
    script: | 
      # this command will will shutdown the existing simulators to save on resources.
      xcrun simctl shutdown all
      #  
      # create new simulator with specified configuration, you can run 
      # 'xcrun simctl list' - to check the list of available simulator configurations
      TEST_DEVICE=$(xcrun simctl create test-device com.apple.CoreSimulator.SimDeviceType.iPhone-11 com.apple.CoreSimulator.SimRuntime.iOS-15-0)
      #
      # boot the newly created simulator
      xcrun simctl boot $TEST_DEVICE
      #
      # run the actual test command
      flutter -d $TEST_DEVICE test integration_test

RESULTS:

Invalid runtime: com.apple.CoreSimulator.SimRuntime.iOS-15-0
Boot a device or device pair.
Usage: simctl boot <device> [--arch=<arch>] [--disabledJob=<job>] [--enabledJob=<job>]

	--arch=<arch>           Specify the architecture to use when booting the simulator (eg: 'arm64' or 'x86_64')
	--disabledJob=<job>     Disables the given launchd job. Multiple jobs can be disabled by passing multiple flags.
	--enabledJob=<job>      Enables the given launchd job when it would normally be disabled.
	                        Multiple jobs can be enabled by passing multiple flags.

If you want to set environment variables in the resulting environment, set them in the calling environment with a SIMCTL_CHILD_ prefix.

Thank you! :)

[danielccdk]

Would really like to set it to a iPhone 15 Pro Max and iOS 17.5 but it doesn't work at all

[dam-ease]

Hello @danielccdk,
Can you try running xcrun simctl list to see the list of available configurations? You can check out the documentation here to confirm the steps for configuration was followed accordingly on your end.
Also, kindly share the build ID for this build if you run into an issue again

[dam-ease]

Per the issue with the integration test failing only on Codemagic, please can you try running the exact command you have in codemagic locally as well?

[danielccdk]

OK...
So I was able to boot up the exact same simulator in codemagic and test against that.
Even tho.. it is NOT visible in the virtual machine. But even if I use the iOS 18 simulator, (which IS visible in the virtual machine) I get the SAME failure:

flutter: Warning: A call to tap() with finder "Found 1 widget with text "Save": [
flutter:   Text("Save", dependencies: [DefaultSelectionStyle, DefaultTextStyle, MediaQuery]),
flutter: ]" derived an Offset (Offset(340.6, 472.0)) that would not hit test on the specified widget.
flutter: Maybe the widget is actually off-screen, or another widget is obscuring it, or the widget cannot receive pointer events.
flutter: The finder corresponds to this RenderBox: RenderParagraph#17c43 relayoutBoundary=up24
flutter: The hit test result at that offset is: HitTestResult(RenderDecoratedBox#fa6ed@Offset(340.6, 1.0), RenderConstrainedBox#07861@Offset(340.6, 1.0), RenderAnimatedOpacity#eccf9@Offset(340.6, 1.0), RenderExcludeSemantics#f16ff@Offset(340.6, 1.0), RenderIgnorePointer#ca3bc@Offset(340.6, 1.0), RenderStack#b222d@Offset(340.6, 1.0), RenderAnimatedSize#1d6b6@Offset(340.6, 1.0), RenderClipRect#e5e59@Offset(340.6, 1.0), RenderFlex#25163@Offset(340.6, 1.0), _RenderInkFeatures#a5de6@Offset(340.6, 1.0), RenderPhysicalModel#f9e60@Offset(340.6, 1.0), RenderStack#3025d@Offset(340.6, 472.0), _RenderTheater#d5d00@Offset(340.6, 472.0), RenderAbsorbPointer#9b6a7@Offset(340.6, 472.0), RenderPointerListener#ec730@Offset(340.6, 472.0), RenderSemanticsAnnotations#1fcd0@Offset(340.6, 472.0), RenderCustomPaint#70f5a@Offset(340.6, 472.0), RenderSemanticsAnnotations#8b134@Offset(340.6, 472.0), RenderSemanticsAnnotations#f64fa@Offset(340.6, 472.0), RenderTapRegionSurface#0d13d@Offset(340.6, 472.0), RenderSemanticsAn<…>
flutter: #0      WidgetController._getElementPoint (package:flutter_test/src/controller.dart:1941:25)
flutter: #1      WidgetController.getCenter (package:flutter_test/src/controller.dart:1792:12)
flutter: #2      WidgetController.tap (package:flutter_test/src/controller.dart:1040:18)
flutter: #3      main.<anonymous closure>.<anonymous closure> (file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart:62:20)
flutter: <asynchronous suspension>
flutter: #4      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
flutter: <asynchronous suspension>
flutter: #5      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
flutter: <asynchronous suspension>
flutter: #6      TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:824:12)
flutter: <asynchronous suspension>
flutter: To silence this warning, pass "warnIfMissed: false" to "tap()".
flutter: To make this warning fatal, set WidgetController.hitTestWarningShouldBeFatal to true.
flutter:
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure was thrown running a test:
flutter: Expected: exactly one matching candidate
flutter:   Actual: _TextWidgetFinder:<Found 0 widgets with text "2017 Jeep Wrangler": []>
flutter:    Which: means none were found but one was expected
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4      main.<anonymous closure>.<anonymous closure> (file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart:71:7)
flutter: <asynchronous suspension>
flutter: #5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
flutter: <asynchronous suspension>
flutter: #6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
flutter: <asynchronous suspension>
flutter: #7      TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:824:12)
flutter: <asynchronous suspension>
flutter:
flutter: This was caught by the test expectation on the following line:
flutter:   file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart line 71
flutter: The test description was:
flutter:   New appraisal is instantly included in appraisals list view
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 01:08 +1: add appraisal test New appraisal is instantly included in appraisals list view [E]
flutter:   Test failed. See exception logs above.
  The test description was: New appraisal is instantly included in appraisals list view
flutter:
flutter: 01:08 +1 -1: (tearDownAll)
Failure Details:
Failure in method: New appraisal is instantly included in appraisals list view
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following TestFailure was thrown running a test:
Expected: exactly one matching candidate
  Actual: _TextWidgetFinder:<Found 0 widgets with text "2017 Jeep
Wrangler": []>
   Which: means none were found but one was expected

When the exception was thrown, this was the stack:
#4      main.<anonymous closure>.<anonymous closure> (file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart:71:7)
<asynchronous suspension>
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
<asynchronous suspension>
#6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
<asynchronous suspension>
#7      TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:824:12)
<asynchronous suspension>

This was caught by the test expectation on the following line:
  file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart line 71
The test description was:
  New appraisal is instantly included in appraisals list view
═════════════════════════════════════════════════════════════════

end of failure 1



flutter: 01:09 +2 -1: Some tests failed.

ALL my integration tests ALWAYS pass locally.

Here is my codemagic.yaml script...

workflows:
  integration_test_workflow:
    environment:
      flutter: "3.24.3"
      xcode: "15.4"
    scripts:
      - name: Integration Tests 
        script: |
          xcrun simctl shutdown all
          #  
          # create new simulator with specified configuration, you can run 
          # 'xcrun simctl list' - to check the list of available simulator configurations
          TEST_DEVICE=$(xcrun simctl create test-device com.apple.CoreSimulator.SimDeviceType.iPhone-15 com.apple.CoreSimulator.SimRuntime.iOS-17-5)
          #
          # boot the newly created simulator
          xcrun simctl boot $TEST_DEVICE
          #
    
          # Run the integration test using flutter drive and define the testing environment variable
          flutter drive \
            --driver=integration_test/driver/driver.dart \
            --target=integration_test/test_master.dart \
            --dart-define=ENV=testing`

Locally I obviously run:

flutter drive --driver=integration_test/driver/driver.dart --target=integration_test/test_master.dart

I'm a bit befuddled how a consistently passing set of tests are failing IN codemagic.

My only thinking is:
I HAVE noticed that the permission handler for iOS (when the app asks for permission to send notifications) is difficult for Flutter. Android it's easy. But despite trying tons of commands, the system dialog to request permission for push notifications is always on the screen. This doesn't appear to be a problem locally. Not sure if that's the case with codemagic?

I think I may pursue this.
Thanks for any help!

[danielccdk]

I removed the push notifications. But STILL....

flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure was thrown running a test:
flutter: Expected: exactly one matching candidate
flutter:   Actual: _TextWidgetFinder:<Found 0 widgets with text "2017 Jeep Wrangler": []>
flutter:    Which: means none were found but one was expected
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4      main.<anonymous closure>.<anonymous closure> (file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart:66:7)
flutter: <asynchronous suspension>
flutter: #5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
flutter: <asynchronous suspension>
flutter: #6      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
flutter: <asynchronous suspension>
flutter: #7      TestWidgetsFlutterBinding._createTestCompletionHandler.<anonymous closure> (package:flutter_test/src/binding.dart:824:12)
flutter: <asynchronous suspension>
flutter:
flutter: This was caught by the test expectation on the following line:
flutter:   file:///Users/builder/clone/integration_test/appraisals/add_appraisal_test.dart line 66
flutter: The test description was:
flutter:   New appraisal is instantly included in appraisals list view
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 01:23 +1: add appraisal test New appraisal is instantly included in appraisals list view [E]
flutter:   Test failed. See exception logs above.
  The test description was: New appraisal is instantly included in appraisals list view

When you can CLEARLY see that the text is on screen and, again, these tests all pass locally.

Genuinely a bit befuddled as this just doesn't make sense...

[dam-ease]

Hello @danielccdk,
Thanks for your response.
I see similar issues in the flutter repo which might suggest an issue in flutter e.g. flutter/flutter#140534 but seeing that it doesn't reproduce locally is where is gets a bit tricky. Are you by any chance making use of OverlayPortal in a ListView?
Also, I see in your command that you are setting a --dart-define=ENV=testing, not sure what this does in your project but do you have this configured locally as well?

[dam-ease]

Hello @danielccdk,
Any update for me on this?If there is anything else I can assist you with, please don't hesitate to let me know.