Skip to content

ci: Use Cirrus Runners for iOS integration tests #4951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Flash0ver wants to merge 26 commits into from
Closed

ci: Use Cirrus Runners for iOS integration tests #4951

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
1df212b
ci: Use Cirrus Runners for iOS integration tests
Flash0ver Feb 23, 2026
192b825
test: add deliberate failure
Flash0ver Feb 24, 2026
25b97b3
Apply suggestions from code review
Flash0ver Feb 24, 2026
04a5b89
ci: Reorder matrix and fix Xcode image tag for net10.0
Flash0ver Feb 24, 2026
7f51356
ci: Add FIFO concurrency groups for Cirrus iOS runners
Flash0ver Feb 24, 2026
175a733
ci: Install PowerShell via dotnet global tool on Cirrus runners
Flash0ver Feb 24, 2026
ebc5740
ci: Add dotnet tools dir to PATH after installing PowerShell
Flash0ver Feb 24, 2026
aff5bf1
build: Update XHarness CLI to 10.0.0-prerelease.25622.2
Flash0ver Feb 25, 2026
841dbc3
ci: Select correct Xcode version before building iOS test app
Flash0ver Feb 25, 2026
7d8d1ed
cI: add debug info
Flash0ver Feb 26, 2026
3776ef8
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
6ec1266
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
f62fba8
Merge branch 'main' into ci/cirrus-runners-for-integration-tests-on-m…
Flash0ver Mar 2, 2026
23897f2
test: remove deliberate failure
Flash0ver Mar 2, 2026
f5cb993
docs: update CSPROJ comment
Flash0ver Mar 2, 2026
b68bb19
ci: normalize iOS tests
Flash0ver Mar 2, 2026
2746f32
revert: cI: add debug info
Flash0ver Mar 2, 2026
2405b8e
ci: change cirrus runner image xcode version
Flash0ver Mar 2, 2026
847e5e8
ci: fix missing Xcode version
Flash0ver Mar 2, 2026
ba35dc4
ci: make Xcode version overwritable
Flash0ver Mar 2, 2026
c294951
ci: Install Pester on Cirrus Runners
Flash0ver Mar 2, 2026
ea37662
ci: change cirrus runner image xcode version
Flash0ver Mar 3, 2026
bec18f0
test: update iOS Device-TestApp to available Xcode version
Flash0ver Mar 3, 2026
aa77b0c
ci: trust PSGallery repository
Flash0ver Mar 3, 2026
d638ee1
ci: omit MinimumVersion of Pester to mimic GitHub-hosted images
Flash0ver Mar 3, 2026
d4cf33f
ci: restore previous behavior for CI on GitHub Actions
Flash0ver Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 39 additions & 11 deletions .github/workflows/device-tests-ios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,18 @@ on:

jobs:
ios-tests:
runs-on: macos-15
name: iOS Tests (${{ matrix.tfm }})
strategy:
fail-fast: false
matrix:
include:
- tfm: net9.0
xcode: "26.0.1"
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.0.1,runner_concurrency_group=${{ github.run_id }}", "runner_group_id:10"]
- tfm: net10.0
xcode: "26.2"
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2,runner_concurrency_group=${{ github.run_id }}", "runner_group_id:10"]
runs-on: ${{ matrix.runs-on }}
env:
DOTNET_CLI_TELEMETRY_OPTOUT: 1
DOTNET_NOLOGO: 1
Expand All @@ -71,40 +82,57 @@ jobs:
- name: Setup Environment
uses: ./.github/actions/environment

- name: Install PowerShell
run: |
which pwsh || (dotnet tool install --global PowerShell && echo "$HOME/.dotnet/tools" >> "$GITHUB_PATH")

- name: Select Xcode
run: sudo xcode-select --switch /Applications/Xcode_${{ matrix.xcode }}.app

- name: Check Rosetta
run: |
arch -x86_64 /usr/bin/true && echo "Rosetta: yes" || echo "Rosetta: no"

- name: List Simulator Runtimes
run: xcrun simctl list runtimes -j
Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Debug Info

Should we remove these two steps again, as we added them for debugging?
Or could at least the 2nd one be generally helpful?
Or maybe make them conditional when "debug logging" is enabled in CI?

Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd throw it away. This just helped to debug why tests refused to run on iossimulator-x64 (no x64 runtime available).


- name: Build iOS Test App
run: pwsh ./scripts/device-test.ps1 ios -Build
run: pwsh ./scripts/device-test.ps1 ios -Build -Tfm ${{ matrix.tfm }}

- name: Run Tests
id: first-test-run
continue-on-error: true
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run -Verbose
Copy link
Copy Markdown

@cursor cursor bot Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Environment action hardcodes Xcode 26.2 breaking net9.0 runner

High Severity

The Setup Environment step (using .github/actions/environment) runs before the Select Xcode step. The environment action hardcodes sudo xcode-select --switch /Applications/Xcode_26.2.app for all macOS runners. On the net9.0 Cirrus image (macos-tahoe-xcode:26.0.1), Xcode 26.2 is not installed — only 26.0.1 is available. This causes the environment setup to fail before the matrix-aware Select Xcode step gets a chance to set the correct version, completely blocking the net9.0 job.

Additional Locations (1)

Fix in Cursor Fix in Web

Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Mar 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed via ba35dc4 by accepting an optional xcode-version input

run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }} -Verbose

- name: Retry Tests (if previous failed to run)
if: steps.first-test-run.outcome == 'failure'
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run -Verbose
run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }} -Verbose

- name: Checkout github-workflows
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: getsentry/github-workflows
ref: 747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
path: modules/github-workflows

- name: Run Integration Tests
id: first-integration-test-run
continue-on-error: true
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Retry Integration Tests (if previous failed to run)
if: steps.first-integration-test-run.outcome == 'failure'
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Upload results
if: success() || failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: device-test-ios-results
name: device-test-ios-results-${{ matrix.tfm }}
path: |
test_output
integration-test/mobile-app/test_output
16 changes: 11 additions & 5 deletions integration-test/ios.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
param(
[string] $dotnet_version = "net10.0"
)

# This file contains test cases for https://pester.dev/
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
Expand All @@ -11,12 +15,14 @@ BeforeDiscovery {
$script:simulator = Get-IosSimulatorUdid -PreferredStates @('Booted')
}

$ios_tpv = switch ($dotnet_version) {
'net9.0' { '26.0' }
default { '26.2' }
}

Describe 'iOS app (<tfm>, <configuration>)' -ForEach @(
# Note: we can't run against net10 and net9 becaus .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
# The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
# net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
@{ tfm = "net10.0-ios26.2"; configuration = "Release" }
@{ tfm = "net10.0-ios26.2"; configuration = "Debug" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Release" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Debug" }
Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Refactor

Is there a "nicer" way to describe this test matrix?

Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can somehow make it work, I'd prefer a simple static matrix:

@{ tfm = "net9.0-ios26.0"; configuration = "Release" }
@{ tfm = "net9.0-ios26.0"; configuration = "Debug" }
@{ tfm = "net10.0-ios26.2"; configuration = "Release" }
@{ tfm = "net10.0-ios26.2"; configuration = "Debug" }

and then somehow filter what we run from the CI. I'm not sure how to do that with Pwsh/Pester from the command line, but the Pester extension for VS Code is able to run specific matrix configurations, so it should be somehow possible... 🤔

Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image image

) -Skip:(-not $script:simulator) {
BeforeAll {
. $PSScriptRoot/../scripts/device-test-utils.ps1
Expand Down
2 changes: 1 addition & 1 deletion integration-test/net9-maui/Sentry.Maui.Device.IntegrationTestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise.
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios26.2</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios26.0;net10.0-ios26.2</TargetFrameworks>

<OutputType>Exe</OutputType>
<RootNamespace>Sentry.Maui.Device.IntegrationTestApp</RootNamespace>
Expand Down
11 changes: 4 additions & 7 deletions test/Sentry.Maui.Device.TestApp/Sentry.Maui.Device.TestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@

<PropertyGroup>
<TargetFrameworks>$(TargetFrameworks);net9.0-android;net10.0-android</TargetFrameworks>
<!-- Note: we can't run against net9-ios because .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios;net10.0-ios</TargetFrameworks>
<DefineConstants Condition="'$(EnableMauiDeviceTestVisualRunner)' == 'true'">$(DefineConstants);VISUAL_RUNNER</DefineConstants>
</PropertyGroup>

Expand Down Expand Up @@ -36,9 +32,10 @@

<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">15.0</SupportedOSPlatformVersion>
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion>
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
<!-- Pin target iOS version to the Xcode version of the respective CI runner image.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise. -->
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">26.2</TargetPlatformVersion>
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net9.0-ios'">26.0</TargetPlatformVersion>
Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Mar 2, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: iOS 26 & Xcode 26 vs iOS 18 & Xcode 16

.NET 9 supports up to Xcode 26.0.

Should we use that version ... or "our lowest" (ios18.0) version instead ... see:

sentry-dotnet/Directory.Build.props

Lines 34 to 35 in 815d9fb

<LatestIosTfm>net10.0-ios26</LatestIosTfm>
<PreviousIosTfm>net9.0-ios18.0</PreviousIosTfm>

Where we would need to - likewise - change the CI runner to Xcode 16.4.

Which was actually the original status before updating the Integration Tests to .NET 10:
https://github.com/getsentry/sentry-dotnet/pull/4750/changes#r2771172008
Or is it better to actually test against the latest (last) supported version for .NET MAUI 9.0?

<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net10.0-ios'">26.2</TargetPlatformVersion>
<EmbedAssembliesIntoApk>true</EmbedAssembliesIntoApk>

<!-- Avoid errors of Sentry.Testing not being a self-contained executable, while including it in a self-contained executable (this).
Expand Down
Loading