dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1118981	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49409
1119033	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50164
1118727	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49682
1117495	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50131
1117194	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1116976	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49531
1116870	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50108
1116651	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50108
1116520	dotnet/sdk	Run TestBuild Tests	Log	#50101
2766637	dotnet-sdk	Run TestBuild Tests	Log	#51985
2766375	dotnet-sdk	🟣 Run TestBuild Tests	Log
1115691	dotnet/sdk	Run TestBuild Tests	Log	#50085
1115347	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50009
1115184	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50073
1115134	dotnet/sdk	Run TestBuild Tests	Log	#50076
1114999	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50073
1114823	dotnet/sdk	Run TestBuild Tests	Log	#49942
1114702	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50063
1113877	dotnet/sdk	Run TestBuild Tests	Log	#49832
1113899	dotnet/sdk	Run TestBuild Tests	Log	#49958
1113881	dotnet/sdk	Run TestBuild Tests	Log	#49843
1113696	dotnet/sdk	Run TestBuild Tests	Log	#49832
2764719	dotnet-sdk	Run TestBuild Tests	Log	#51985
1113053	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1113136	dotnet/sdk	Run TestBuild Tests	Log	#50039
1113022	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49556
1112425	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49556
1110213	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49954
1112396	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49556
1111690	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50002
1111427	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49531
1110748	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49531
2761722	dotnet-sdk	Run FullFramework Tests	Log	#51985
1109785	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50002
1109717	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50002
1109338	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49991
1109006	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49974
1108155	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49947
1107766	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49974
1106456	dotnet/sdk	Run FullFramework Tests	Log	#49946
1106341	dotnet/sdk	Run TestBuild Tests	Log	#49942
1105212	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
1104821	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49740
1104469	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49857
1102342	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49901
1101555	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
1101651	dotnet/sdk	Run TestBuild Tests	Log	#49891
1099538	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1099431	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49544
1101273	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
1095430	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49417
1099618	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
2753126	dotnet-sdk	Run TestBuild Tests	Log	#51708
1099251	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
1099206	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49740
1098820	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49740
1097882	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49634
1098687	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49740
2752661	dotnet-sdk	Run FullFramework Tests	Log	#51694
1098287	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49409
2752492	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1098299	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49847
1098231	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49848
1098338	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49798
1097069	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49824
1098131	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49838
1098153	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49819
1098112	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#48699
1098206	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49800
1098073	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49409
1097859	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49800
1097911	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49841
1097433	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49798
1097831	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49838
2752069	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1097735	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49740
1097605	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49635
1097619	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49802
1097645	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49800
1097615	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49838
2752016	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1097607	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49642
1097447	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49409
1097434	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49802
1096317	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49806
2751857	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1097318	dotnet/sdk	Run TestBuild Tests	Log	#49832
1097312	dotnet/sdk	Run TestBuild Tests	Log	#49831
2751672	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
2751758	dotnet-sdk	🟣 Run ContainerBased Tests (azureLinux30Net10BuildAmd64)	Log
1096992	dotnet/sdk	🟣 Run FullFramework Tests	Log	#49531
1096993	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#48819
1096991	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49544
1096990	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49541
1096885	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49818
1096042	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49798
1096679	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49697
1095795	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49746
1096725	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49814
1096543	dotnet/sdk	🟣 Run ContainerBased Tests	Log	#49810
Displaying 100 of 162 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	26	162

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.