slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1343509	dotnet/maui	Controls Layout	Log
2931049	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#59239
1344037	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125807
1343386	dotnet/maui	Controls Shell	Log
1343948	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125793
1343334	dotnet/maui	Controls CollectionView	Log
1343817	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125762
1343305	dotnet/maui	Controls SearchBar,Shape,Slider	Log
1343813	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1343745	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53539
1343562	dotnet/maui	Controls CollectionView	Log
1343693	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#125507
1343712	dotnet/aspnetcore	Tests: macOS	Log
1343138	dotnet/maui	Controls Layout	Log
1343488	dotnet/maui	Controls Layout	Log	dotnet/maui#26475
2930761	dotnet-maui	Build macOS	Log
1343088	dotnet/maui	Controls CollectionView	Log
1343479	dotnet/runtime	osx-arm64 release CrossAOT_Mono crossaot	Log	dotnet/runtime#125782
1343529	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#16613
1343557	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124893
2930762	dotnet-maui	Controls CollectionView	Log
1343490	dotnet/runtime	osx-arm64 release CrossAOT_Mono crossaot	Log	dotnet/runtime#125785
2930750	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
2930751	dotnet-maui	Controls CollectionView	Log
1342931	dotnet/maui	Controls Shell	Log
2930727	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
2930728	dotnet-maui	Controls Shell	Log
2930701	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
2930684	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1343310	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53517
2930685	dotnet-maui	Controls CollectionView	Log
2930646	dotnet-maui	Build macOS	Log
1343195	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53350
2930647	dotnet-maui	Controls Layout	Log
1343131	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#52930
1342704	dotnet/maui	Controls Shell	Log
2930612	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1343032	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65648
2930583	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper 8.0	Log
1342560	dotnet/maui	Controls Layout	Log	dotnet/maui#33469
1342713	dotnet/maui	Controls Layout	Log	dotnet/maui#34453
1342577	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34510
1342562	dotnet/maui	Controls Layout	Log	dotnet/maui#32794
1342633	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65870
1342573	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64964
1342319	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
2930214	dotnet-sdk	Darwin	Log	#59224
1342468	dotnet/aspnetcore	Tests: macOS	Log
2930213	dotnet-sdk	Darwin	Log	#59223
2930309	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1342414	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2930286	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1342381	dotnet/aspnetcore	Tests: macOS	Log
1342208	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1341868	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#125512
1342190	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53424
1342173	dotnet/aspnetcore	Tests: macOS	Log
1341246	dotnet/maui	Controls WebView	Log
1341771	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53187
1342131	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65864
1342107	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65856
2930194	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#57892
1341975	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65846
1341650	dotnet/maui	Controls Layout	Log
1341023	dotnet/maui	Controls Shell	Log	dotnet/maui#29952
1341738	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#124770
1341757	dotnet/aspnetcore	Tests: macOS	Log
2930080	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#58741
2929887	dotnet-maui	Controls (v18.4) SoftInput,Stepper,Switch,SwipeView	Log
2929995	dotnet-maui	Controls (vlatest) Label	Log
1341633	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2929912	dotnet-maui	RunOniOS_BlazorRelease x64	Log
2929942	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#59204
1341447	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123990
1341324	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124637
2929742	dotnet-maui	Controls Page,Performance,Picker,ProgressBar	Log
1341208	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125646
1341199	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121065
1341185	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#125289
1336777	dotnet/machinelearning	MacOS_cross_arm64 Debug_Build	Log	dotnet/machinelearning#7566
1341124	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65728
1341083	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53515
1341056	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65816
1337915	dotnet/maui	Controls Shell	Log	dotnet/maui#34107
1340796	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#125520
1340818	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#125283
1340658	dotnet/maui	Controls Layout	Log
1340773	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#125260
1340936	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1531
1340912	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125524
1340921	dotnet/aspnetcore	Tests: macOS	Log
1340870	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123874
1339283	dotnet/maui	Controls Shell	Log
1340723	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#125656
2929525	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1340649	dotnet/aspnetcore	Tests: macOS	Log
2929526	dotnet-maui	Controls Shell	Log
1340495	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33908
1340625	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124604
1325908	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1541
Displaying 100 of 862 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
49	228	809

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.