dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1286936	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52648
2899627	dotnet-sdk	Run TestBuild Tests	Log	#57694
1286713	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52897
1286834	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52923
2899510	dotnet-sdk	Run TestBuild Tests	Log	#57694
2899410	dotnet-sdk	Run TestBuild Tests	Log	#57694
1286410	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52897
1285710	dotnet/sdk	Run TestBuild Tests	Log	#52907
1285141	dotnet/sdk	Run TestBuild Tests	Log	#52902
1284292	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52648
1284297	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52897
2897760	dotnet-sdk	Run FullFramework Tests	Log	#57654
1283528	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52581
1283494	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52711
1283453	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52881
1283161	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52581
1282513	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52711
2896843	dotnet-sdk	Run FullFramework Tests	Log	#56922
2896788	dotnet-sdk	Run TestBuild Tests	Log	#56922
1282031	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52581
1282061	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52836
1282041	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52816
2896557	dotnet-sdk	Run TestBuild Tests	Log	#56922
1281867	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52858
1281874	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52860
1281749	dotnet/sdk	Run TestBuild Tests	Log	#52857
1281518	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52554
1281072	dotnet/sdk	Run TestBuild Tests	Log	#52846
2896127	dotnet-sdk	Run TestBuild Tests	Log	#56922
2896038	dotnet-sdk	Run FullFramework Tests	Log	#56922
1280788	dotnet/sdk	Run FullFramework Tests	Log	#52843
2895899	dotnet-sdk	Run TestBuild Tests	Log	#56922
2895743	dotnet-sdk	Run TestBuild Tests	Log	#57546
2895744	dotnet-sdk	Run FullFramework Tests	Log	#56922
1279322	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52711
1278295	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52808
1277672	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52711
1277270	dotnet/sdk	Run TestBuild Tests	Log	#52591
1277271	dotnet/sdk	Run TestBuild Tests	Log	#52592
1277273	dotnet/sdk	Run TestBuild Tests	Log	#52594
1277262	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52596
1277261	dotnet/sdk	Run TestBuild Tests	Log	#52786
1277303	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52798
1275993	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52711
1275892	dotnet/sdk	Run TestBuild Tests	Log	#52606
1275884	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52711
1275565	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52760
1275067	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52760
1274976	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49556
1274744	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52760
1274278	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52760
1274202	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52664
1274032	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52711
1274067	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52752
1273904	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52752
1273779	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52044
1273833	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52742
1273601	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52713
1273636	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52044
1270343	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52405
1273029	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52722
1273026	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52596
1272968	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52742
1273015	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52743
1272815	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52664
1272286	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52664
1272255	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52710
1272361	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52730
1272124	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52044
1272214	dotnet/sdk	Run TestBuild Tests	Log	#52735
1271998	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52733
1271958	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52731
1271972	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52710
1271829	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52729
1271802	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52710
1271746	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52585
1271508	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52722
1271728	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52729
1268564	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52684
1271442	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52596
1271367	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52044
1271566	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52725
1270984	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52709
1270790	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52713
1269810	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52585
1269887	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52597
1268262	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52235
1267931	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52677
1265882	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
1265150	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
2886447	dotnet-sdk	Run FullFramework Tests	Log	#56922
1265178	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52632
1264700	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
1264171	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52585
1264022	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52648
1263944	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52585
1264059	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52597
1263963	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52575
1263817	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52648
1263065	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52638
Displaying 100 of 159 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
9	43	159

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.