Skip to content

Conversation

jpnurmi
Copy link
Collaborator

@jpnurmi jpnurmi commented Sep 24, 2025

Helps verify that crash events are captured as expected after Cocoa and Java SDK updates (and eventually getsentry/sentry-cocoa#6193):

  • CrashType.Managed
  • CrashType.Native
  • NullReferenceException

For example, see

  • 51240026268 (iOS Device Tests -> Run Integration Tests)
  • 51593510741 (Android Device Tests -> Run Integration Tests)

#skip-changelog


Note

Adds MAUI-based Android and iOS integration tests executed on emulators/simulators in CI, with supporting test app, utilities, and workflow updates (including retries and result artifacts).

  • CI/Workflows:
    • Android: run new integration tests on emulator via integration-test/android.Tests.ps1 with retry; checkout submodules; set up Java 17; fetch shared github-workflows; reuse emulator config via YAML anchors; upload both test_output and integration-test/mobile-app/test_output.
    • iOS: add integration tests via integration-test/ios.Tests.ps1 with retry; upload both result paths.
    • Build: include scripts in sparse checkout for publishing builds.
  • Tests/Infrastructure:
    • Add MAUI test app (integration-test/net9-maui/**) targeting net9.0-android35.0 and net9.0-ios18.0 to drive crash scenarios and verify envelopes.
    • New Pester test suites: integration-test/android.Tests.ps1, integration-test/ios.Tests.ps1; shared helpers in integration-test/pester.ps1.
    • Utilities: new scripts/device-test-utils.ps1 (XHarness install, emulator/simulator helpers); scripts/device-test.ps1 now uses these helpers and simplifies simulator selection.
    • Refactor integration-test/common.ps1 to source shared Pester extensions.

Written by Cursor Bugbot for commit 8144e87. This will update automatically on new commits. Configure here.

Copy link

codecov bot commented Sep 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.46%. Comparing base (b3d9b2f) to head (fed73a4).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4559      +/-   ##
==========================================
- Coverage   73.50%   73.46%   -0.04%     
==========================================
  Files         482      482              
  Lines       17678    17678              
  Branches     3493     3493              
==========================================
- Hits        12994    12988       -6     
- Misses       3796     3800       +4     
- Partials      888      890       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jpnurmi
Copy link
Collaborator Author

jpnurmi commented Sep 25, 2025

@sentry review

@jamescrosswell
Copy link
Collaborator

Running integration-test/ios.Tests.ps1, I am getting the same system crash dialog you mentioned @jpnurmi... I'm guessing (since all of this passes on CI) that it's not an issue when running headless on CI.

However, locally everything seems to grind to a halt and the tests don't complete after xcrun simctl launch --console --terminate-running-process is executed.

I'll send a full console log via DM to see if you've got any ideas.

@jpnurmi
Copy link
Collaborator Author

jpnurmi commented Oct 2, 2025

Is there anything special I should be doing locally before running the script? I do have the emulator running and I see the app loading (for quite a while).

It should be enough to have the emulator up and running. I suspect something with the host environment because it works for me on both macOS and Linux, and it passes in the CI as well. Could you try applying Android SDK/system image/tool/emulator updates and see if that helps? I suspect there are updates available because you mentioned you don't have Pixel 9 on the list. 🙂

However, locally everything seems to grind to a halt and the tests don't complete after xcrun simctl launch --console --terminate-running-process is executed.

The hang looks random and is not related to any specific change, after all. Might the infamous Environment.Exit(0) call (#4559 (comment)). I'll try to find a way to make it quit in a way that XHarness understands.

@jpnurmi jpnurmi marked this pull request as draft October 2, 2025 16:13
- just pass one test arg
- minimize logging (Swift print deadlocks???)
- ios: switch from xharness to plain xcrun simctl
Allows running Android integration tests on Windows
To avoid requiring `adb` in `PATH`
Now `pwsh integration-test/android.Tests.ps1` device/emulator discovery
works without adb in PATH or ANDROID_SERIAL explicitly defined.
@jpnurmi jpnurmi marked this pull request as ready for review October 6, 2025 08:46
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

# by Android Device Tests, or manually when testing locally. This avoids
# slowing down non-Device Test CI builds further.
Install-XHarness
$script:emulator = Get-AndroidEmulatorId
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL about $script scoped variables in PWSH!

Copy link
Collaborator

@jamescrosswell jamescrosswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one non-blocking suggestion, but otherwise LGTM 🎉

cursor[bot]

This comment was marked as outdated.

Technically, less correct but doesn't fail even if port forwarding had
failed. This way, we can keep it simple and don't need to track it
separately.
@jpnurmi
Copy link
Collaborator Author

jpnurmi commented Oct 7, 2025

Thanks for testing and helping!

@jpnurmi jpnurmi merged commit e58a8f1 into main Oct 7, 2025
32 checks passed
@jpnurmi jpnurmi deleted the test/ios-integration branch October 7, 2025 11:13
alexsohn1126 pushed a commit that referenced this pull request Oct 11, 2025
Co-authored-by: Stefan Pölz <[email protected]>
Co-authored-by: James Crosswell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants