Skip to content

ci: Use Cirrus Runners for iOS integration tests #4951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Flash0ver wants to merge 26 commits into from
Closed

ci: Use Cirrus Runners for iOS integration tests #4951

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
1df212b
ci: Use Cirrus Runners for iOS integration tests
Flash0ver Feb 23, 2026
192b825
test: add deliberate failure
Flash0ver Feb 24, 2026
25b97b3
Apply suggestions from code review
Flash0ver Feb 24, 2026
04a5b89
ci: Reorder matrix and fix Xcode image tag for net10.0
Flash0ver Feb 24, 2026
7f51356
ci: Add FIFO concurrency groups for Cirrus iOS runners
Flash0ver Feb 24, 2026
175a733
ci: Install PowerShell via dotnet global tool on Cirrus runners
Flash0ver Feb 24, 2026
ebc5740
ci: Add dotnet tools dir to PATH after installing PowerShell
Flash0ver Feb 24, 2026
aff5bf1
build: Update XHarness CLI to 10.0.0-prerelease.25622.2
Flash0ver Feb 25, 2026
841dbc3
ci: Select correct Xcode version before building iOS test app
Flash0ver Feb 25, 2026
7d8d1ed
cI: add debug info
Flash0ver Feb 26, 2026
3776ef8
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
6ec1266
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
f62fba8
Merge branch 'main' into ci/cirrus-runners-for-integration-tests-on-m…
Flash0ver Mar 2, 2026
23897f2
test: remove deliberate failure
Flash0ver Mar 2, 2026
f5cb993
docs: update CSPROJ comment
Flash0ver Mar 2, 2026
b68bb19
ci: normalize iOS tests
Flash0ver Mar 2, 2026
2746f32
revert: cI: add debug info
Flash0ver Mar 2, 2026
2405b8e
ci: change cirrus runner image xcode version
Flash0ver Mar 2, 2026
847e5e8
ci: fix missing Xcode version
Flash0ver Mar 2, 2026
ba35dc4
ci: make Xcode version overwritable
Flash0ver Mar 2, 2026
c294951
ci: Install Pester on Cirrus Runners
Flash0ver Mar 2, 2026
ea37662
ci: change cirrus runner image xcode version
Flash0ver Mar 3, 2026
bec18f0
test: update iOS Device-TestApp to available Xcode version
Flash0ver Mar 3, 2026
aa77b0c
ci: trust PSGallery repository
Flash0ver Mar 3, 2026
d638ee1
ci: omit MinimumVersion of Pester to mimic GitHub-hosted images
Flash0ver Mar 3, 2026
d4cf33f
ci: restore previous behavior for CI on GitHub Actions
Flash0ver Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 32 additions & 11 deletions .github/workflows/device-tests-ios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,18 @@ on:

jobs:
ios-tests:
runs-on: macos-15
name: iOS Tests (${{ matrix.tfm }})
strategy:
fail-fast: false
matrix:
include:
- tfm: net9.0
xcode: "26.0.1"
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.0.1,runner_concurrency_group=${{ github.run_id }}", "runner_group_id:10"]
- tfm: net10.0
xcode: "26.2"
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2,runner_concurrency_group=${{ github.run_id }}", "runner_group_id:10"]
runs-on: ${{ matrix.runs-on }}
env:
DOTNET_CLI_TELEMETRY_OPTOUT: 1
DOTNET_NOLOGO: 1
Expand All @@ -71,40 +82,50 @@ jobs:
- name: Setup Environment
uses: ./.github/actions/environment

- name: Install PowerShell
shell: bash
run: which pwsh || (dotnet tool install --global PowerShell && echo "$HOME/.dotnet/tools" >> "$GITHUB_PATH")

- name: Select Xcode
run: sudo xcode-select --switch /Applications/Xcode_${{ matrix.xcode }}.app

- name: Build iOS Test App
run: pwsh ./scripts/device-test.ps1 ios -Build
run: pwsh ./scripts/device-test.ps1 ios -Build -Tfm ${{ matrix.tfm }}

- name: Run Tests
id: first-test-run
continue-on-error: true
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run
run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }}

- name: Retry Tests (if previous failed to run)
if: steps.first-test-run.outcome == 'failure'
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run
run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }}

- name: Checkout github-workflows
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: getsentry/github-workflows
ref: 747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
path: modules/github-workflows

- name: Run Integration Tests
id: first-integration-test-run
continue-on-error: true
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Retry Integration Tests (if previous failed to run)
if: steps.first-integration-test-run.outcome == 'failure'
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Upload results
if: success() || failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: device-test-ios-results
name: device-test-ios-results-${{ matrix.tfm }}
path: |
test_output
integration-test/mobile-app/test_output
16 changes: 11 additions & 5 deletions integration-test/ios.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
param(
[string] $dotnet_version = "net10.0"
)

# This file contains test cases for https://pester.dev/
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
Expand All @@ -11,12 +15,14 @@ BeforeDiscovery {
$script:simulator = Get-IosSimulatorUdid -PreferredStates @('Booted')
}

$ios_tpv = switch ($dotnet_version) {
'net9.0' { '26.0' }
default { '26.2' }
}

Describe 'iOS app (<tfm>, <configuration>)' -ForEach @(
# Note: we can't run against net10 and net9 becaus .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
# The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
# net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
@{ tfm = "net10.0-ios26.2"; configuration = "Release" }
@{ tfm = "net10.0-ios26.2"; configuration = "Debug" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Release" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Debug" }
Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Refactor

Is there a "nicer" way to describe this test matrix?

Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can somehow make it work, I'd prefer a simple static matrix:

@{ tfm = "net9.0-ios26.0"; configuration = "Release" }
@{ tfm = "net9.0-ios26.0"; configuration = "Debug" }
@{ tfm = "net10.0-ios26.2"; configuration = "Release" }
@{ tfm = "net10.0-ios26.2"; configuration = "Debug" }

and then somehow filter what we run from the CI. I'm not sure how to do that with Pwsh/Pester from the command line, but the Pester extension for VS Code is able to run specific matrix configurations, so it should be somehow possible... 🤔

Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image image

) -Skip:(-not $script:simulator) {
BeforeAll {
. $PSScriptRoot/../scripts/device-test-utils.ps1
Expand Down
2 changes: 1 addition & 1 deletion integration-test/net9-maui/Sentry.Maui.Device.IntegrationTestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise.
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios26.2</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios26.0;net10.0-ios26.2</TargetFrameworks>

<OutputType>Exe</OutputType>
<RootNamespace>Sentry.Maui.Device.IntegrationTestApp</RootNamespace>
Expand Down
2 changes: 1 addition & 1 deletion scripts/device-test-utils.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ function Install-XHarness
{
$CI = Test-Path env:CI
Push-Location ($CI ? $env:RUNNER_TEMP : $IsWindows ? $env:TMP : $IsMacos ? $env:TMPDIR : '/tmp')
dotnet tool install Microsoft.DotNet.XHarness.CLI --global --version '10.0.0-prerelease.25466.1' `
dotnet tool install Microsoft.DotNet.XHarness.CLI --global --version '10.0.0-prerelease.25622.2' `
--add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json
Pop-Location
}
Expand Down
9 changes: 3 additions & 6 deletions test/Sentry.Maui.Device.TestApp/Sentry.Maui.Device.TestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@

<PropertyGroup>
<TargetFrameworks>$(TargetFrameworks);net9.0-android;net10.0-android</TargetFrameworks>
<!-- Note: we can't run against net9-ios because .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios;net10.0-ios</TargetFrameworks>
<DefineConstants Condition="'$(EnableMauiDeviceTestVisualRunner)' == 'true'">$(DefineConstants);VISUAL_RUNNER</DefineConstants>
</PropertyGroup>

Expand Down Expand Up @@ -38,7 +34,8 @@
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion>
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise. -->
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">26.2</TargetPlatformVersion>
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net10.0-ios'">26.2</TargetPlatformVersion>
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net9.0-ios'">26.0</TargetPlatformVersion>
<EmbedAssembliesIntoApk>true</EmbedAssembliesIntoApk>

<!-- Avoid errors of Sentry.Testing not being a self-contained executable, while including it in a self-contained executable (this).
Expand Down
6 changes: 6 additions & 0 deletions test/Sentry.Tests/SentrySdkTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@
_logger = Substitute.ForPartsOf<TestOutputDiagnosticLogger>(testOutputHelper);
}

[Fact]
public void This_Test_Should_Fail()
Copy link
Copy Markdown
Member Author

@Flash0ver Flash0ver Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: remove

I added this to see a test failure in CI ... to double check against a false negative

{
Assert.True(SentrySdk.IsEnabled);

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False
}

Check failure on line 33 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

[Fact]
public void IsEnabled_StartsOffFalse()
{
Expand Down
Loading