dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1362081	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53658
1360283	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52396
2940010	dotnet-sdk	Run TestBuild Tests	Log	#59580
2940009	dotnet-sdk	Run TestBuild Tests	Log	#59579
1358711	dotnet/sdk	🟣 Run FullFramework Tests	Log	#52396
1358515	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53464
1356401	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53614
1356064	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53609
1355481	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1355413	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53616
1354256	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53609
1353757	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53547
1353558	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53602
2936188	dotnet-sdk	Run TestBuild Tests	Log	#59431
1352845	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53597
1352086	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53591
1351238	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53564
1350762	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53555
1350605	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53549
1349180	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53564
1345805	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53555
1348822	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53564
2932676	dotnet-sdk	Run TestBuild Tests	Log	#59217
1346083	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53424
1345945	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1345664	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1345185	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53539
1343910	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53424
1343686	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53424
1343430	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1342528	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53350
1341747	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53517
1341771	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53187
1341135	dotnet/sdk	🟣 Run TestBuild Tests	Log	#52930
1339118	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53487
1339430	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53491
1339347	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53187
1338631	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53487
1338382	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53143
1338399	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1338613	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53484
1338231	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53487
1337763	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1337805	dotnet/sdk	Run TestBuild Tests	Log	#53377
1337450	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53476
1336689	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53477
2927324	dotnet-sdk	Run TestBuild Tests	Log	#58430
1336474	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53271
1336424	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1336065	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53464
1335909	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53466
1335889	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53464
1335872	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53187
1335586	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53187
2926459	dotnet-sdk	Run TestBuild Tests	Log	#58431
1335136	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53457
1335186	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53459
1335087	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1334799	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1334881	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53424
1334614	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53143
1334492	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53181
1334111	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53290
1333930	dotnet/sdk	Run TestBuild Tests	Log	#53437
1333740	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53435
1333596	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53181
1333493	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53290
1333431	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53432
1333427	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53431
1333316	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1333301	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53143
1333365	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53428
1333217	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53290
1332998	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1333120	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53424
1332737	dotnet/sdk	🟣 Run TestBuild Tests	Log	#45637
1332165	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1332127	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53407
1331643	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1331618	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53350
1331328	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53175
1331136	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53290
1328667	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1331214	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53345
1330906	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53394
1330055	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53371
1329991	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53381
1329755	dotnet/sdk	🟣 Run FullFramework Tests	Log	#45637
1329779	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53350
1329629	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53345
1329499	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53371
1329634	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53350
1329577	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53375
1329525	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53372
1329124	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53340
1329001	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53353
1328628	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53290
1327952	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
1327734	dotnet/sdk	🟣 Run FullFramework Tests	Log	#53335
1327798	dotnet/sdk	🟣 Run TestBuild Tests	Log	#53284
Displaying 100 of 145 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	15	145

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.