System.Formats.Nrbf.Tests timeouts

[EgorBo]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=882740
Build error leg or test failing:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "'System.Formats.Nrbf.Tests' END OF WORK ITEM LOG: Command timed out, and was killed",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=882740
Error message validated: ['System.Formats.Nrbf.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/30/2024 4:39:23 PM UTC

Report

Build	Definition	Test	Pull Request
892948	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110554
892884	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110342
892786	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#109826
892767	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110198
892724	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110589
892643	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110619
891862	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110322
892590	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110617
892519	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110330
892495	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110613
892403	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110477
892370	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110612
892361	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110610
890559	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110545
892224	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110477
892221	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110567
892128	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110477
892120	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110589
892104	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110602
891876	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110477
891834	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110554
891774	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110342
891768	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110548
891637	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110589
891613	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110574
891601	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110586
891157	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110566
891492	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110579
891484	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110330
891469	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110576
891445	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#108750
891442	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110197
891430	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110574
891042	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110396
890210	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110531
890941	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110353
890938	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110105
890782	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110557
890774	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#109136
890349	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110494
890662	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110465
890658	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110105
890588	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110540
890581	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110542
890256	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110481
890562	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110551
890471	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110548
890465	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110520
890436	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110518
890418	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110322
890379	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110516
890306	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#109378
890289	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110501
890246	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110536
890226	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110535
890192	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110471
889958	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110528
889951	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110308
889230	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110484
889766	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110522
889597	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110518
889586	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110516
889581	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110501
889212	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110506
889532	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110513
889527	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110481
889444	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110465
889401	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110501
889383	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110501
889250	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110342
889192	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110342
889140	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110033
889095	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110503
888986	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#104906
2597163	dotnet-runtime	System.Formats.Nrbf.Tests.WorkItemExecution
888810	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110485
888893	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110342
888474	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110449
888803	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110481
888785	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110486
888660	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110494
888709	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110033
888706	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110465
888617	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110448
888589	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110491
888566	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110471
888563	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110484
888561	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110476
888553	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110486
888537	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110064
888494	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110477
888424	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#104906
888417	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110484
888408	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110483
888363	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110481
888292	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#108941
888265	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#104906
888192	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110119
888153	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110472
888145	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	#110471
Displaying 100 of 211 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
19	124	211

[EgorBo]

Could be #110194 ? cc @adamsitnik

[am11]

All these failures seem to be exclusive to win-x86. e.g. https://helix.dot.net/api/2019-06-17/jobs/6ec31758-1d44-4d2d-a257-0af23b46ac9a/workitems/System.Formats.Nrbf.Tests/console

[adamsitnik]

Of course I can't repro it locally..

.\build.cmd -c Release -subset clr+libs+libs.tests -arch x86
.\dotnet.cmd build -c Release .\src\libraries\System.Formats.Nrbf\tests\System.Formats.Nrbf.Tests.csproj /t:Test /p:TargetArchitecture=x86

  ----- start Tue 12/03/2024 16:45:01.66 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Release\net10.0\
  "D:\projects\runtime\artifacts\bin\testhost\net10.0-windows-Release-x86\dotnet.exe" exec --runtimeconfig System.Formats.Nrbf.Tests.runtimeconfig.json --depsfile System.Formats.Nrbf.Tests.deps.json C:\Users\adsitnik\.nuget\packages\mi
  crosoft.dotnet.xunitconsolerunner\2.9.2-beta.24570.4\build\..\tools\net\xunit.console.dll System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 125 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
      System.Formats.Nrbf.Tests.EdgeCaseTests.CanReadArrayOfAnySize [SKIP]
        Condition(s) not met: "Is64BitProcess"
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 365, Errors: 0, Failed: 0, Skipped: 1, Time: 12.073s
  ----- end Tue 12/03/2024 16:45:14.71 ----- exit code 0 ----------------------------------------------------------
  ----- start Tue 12/03/2024 16:45:36.03 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Release\net462\
  C:\Users\adsitnik\.nuget\packages\xunit.runner.console\2.9.2\build\..\tools\net472\xunit.console.exe System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 121 of 123 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 363, Errors: 0, Failed: 0, Skipped: 0, Time: 9.918s
  ----- end Tue 12/03/2024 16:45:46.64 ----- exit code 0 ----------------------------------------------------------

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:52.31

.\build.cmd -rc Release -subset clr+libs+libs.tests -arch x86
.\dotnet.cmd build .\src\libraries\System.Formats.Nrbf\tests\System.Formats.Nrbf.Tests.csproj /t:Test /p:TargetArchitecture=x86

  ========================== End custom configuration settings ===============================
  ----- start Tue 12/03/2024 14:05:44.36 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Debug\net10.0\
  "D:\projects\runtime\artifacts\bin\testhost\net10.0-windows-Debug-x86\dotnet.exe" exec --runtimeconfig System.Formats.Nrbf.Tests.runtimeconfig.json --depsfile System.Formats.Nrbf.Tests.deps.json C:\Users\adsitnik\.nuget\packages\micr
  osoft.dotnet.xunitconsolerunner\2.9.2-beta.24570.4\build\..\tools\net\xunit.console.dll System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 125 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
      System.Formats.Nrbf.Tests.EdgeCaseTests.CanReadArrayOfAnySize [SKIP]
        Condition(s) not met: "Is64BitProcess"
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 365, Errors: 0, Failed: 0, Skipped: 1, Time: 21.922s
  ----- end Tue 12/03/2024 14:06:07.86 ----- exit code 0 ----------------------------------------------------------
  ========================== End custom configuration settings ===============================
  ----- start Tue 12/03/2024 14:06:28.71 ===============  To repro directly: =====================================================
  pushd D:\projects\runtime\artifacts\bin\System.Formats.Nrbf.Tests\Debug\net462\
  C:\Users\adsitnik\.nuget\packages\xunit.runner.console\2.9.2\build\..\tools\net472\xunit.console.exe System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Nrbf.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Nrbf.Tests (found 121 of 123 test cases)
    Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [24 threads], stop on fail = off)
    Finished:    System.Formats.Nrbf.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Nrbf.Tests  Total: 363, Errors: 0, Failed: 0, Skipped: 0, Time: 10.364s
  ----- end Tue 12/03/2024 14:06:40.41 ----- exit code 0 ----------------------------------------------------------

I'll try to reproduce it on Win 10

[jkotas]

build.cmd -rc Release

The timeouts occur with x86-coreclr_checked. Try with -rc checked.

parallel test collections = on [24 threads]

The CI machines have much smaller number of cores. This message on CI machines says 4 threads. You may want to restrict the test parallelization to better much the CI conditions.