Add MSBUILDDEBUGONSTART=3 to skip debugging TaskHost child processes (#12679)

## Summary: Add MSBUILDDEBUGONSTART=3 to skip debugging TaskHost
processes

✅ **All tasks completed successfully**

- [x] Explore repository structure and understand current implementation
- [x] Understand how MSBUILDDEBUGONSTART works in XMake.cs and
OutOfProcTaskHost.cs
- [x] Identify that DebugUtils.cs has NodeMode detection capability
- [x] Build repository successfully to verify starting state
- [x] Create helper method IsInTaskHostNode() in DebugUtils.cs
- [x] Update XMake.cs to support new environment variable value "3"
(launches debugger conditionally)
- [x] Update OutOfProcTaskHost.cs to handle value "3" (skips debugging
for TaskHost)
- [x] Add tests for the new IsInTaskHostNode() method
- [x] Build and verify the changes work
- [x] Run tests to ensure no regressions
- [x] Manual verification of the feature
- [x] Code review and address feedback
- [x] **Fixed: Added IsInTaskHostNode() check in XMake.cs to handle
sidecar TaskHost processes**
- [x] **Updated documentation to reference new MSBUILDDEBUGONSTART=3
behavior**
- [x] CodeQL security scan
- [x] Final validation

## Changes Made (42 lines added/modified across 6 files):

1. **src/Shared/Debugging/DebugUtils.cs**: Added `IsInTaskHostNode()`
public method to detect if current process is a TaskHost
2. **src/MSBuild/XMake.cs**: Added case "3" with conditional check -
launches debugger only if NOT in TaskHost mode
3. **src/MSBuildTaskHost/OutOfProcTaskHost.cs**: Added case "3" to skip
debugger launch for TaskHost processes
4. **src/Build.UnitTests/BackEnd/DebugUtils_tests.cs**: Added test to
verify IsInTaskHostNode() works correctly
5.
**documentation/wiki/Building-Testing-and-Debugging-on-.Net-Core-MSBuild.md**:
Updated debugging section with new value "3"
6.
**documentation/wiki/Building-Testing-and-Debugging-on-Full-Framework-MSBuild.md**:
Updated breakpoints section with new value "3"

## How to Use:
Set `MSBUILDDEBUGONSTART=3` before running MSBuild to debug the main
process while skipping both:
- MSBuildTaskHost.exe child processes
- Sidecar TaskHost processes running in MSBuild.exe

**Available values:**
- `1`: Launch debugger for all processes (main + TaskHost)
- `2`: Wait for manual debugger attach for all processes  
- `3` (NEW): Launch debugger for main MSBuild only, skip TaskHost
processes

## Documentation:
✅ Updated in two wiki pages:
- Building-Testing-and-Debugging-on-.Net-Core-MSBuild.md
- Building-Testing-and-Debugging-on-Full-Framework-MSBuild.md

## Security:
✅ No security vulnerabilities detected by CodeQL

## Testing:
✅ All tests pass (2/2 DebugUtils tests)
✅ Build succeeds with no warnings or errors
✅ Manual verification completed

<!-- START COPILOT CODING AGENT SUFFIX -->



<details>

<summary>Original prompt</summary>

> There is the MSBUILDDEBUGONSTART environment variable that launches
debugging at the start when set. However it launches it also for child
processes that inherit the environment (e.g. the Sidecar TaskHost
processes, which I don't want to debug every time). Please add a new
value for the variable that when set first checks if the current process
is a sidecar taskhost and in that case does not launch debugging.


</details>



<!-- START COPILOT CODING AGENT TIPS -->
---

💬 Share your feedback on Copilot coding agent for the chance to win a
$200 gift card! Click
[here](https://survey3.medallia.com/?EAHeSx-AP01bZqG0Ld9QLQ) to start
the survey.

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: JanProvaznik <[email protected]>

Author: Copilot

documentation/wiki/Building-Testing-and-Debugging-on-.Net-Core-MSBuild.md

@@ -40,7 +40,12 @@ The best way to get .NET Core MSBuild is by installing the [.NET Core SDK](https
 
 ## Wait in Main
 
-Set the environment variable `MSBUILDDEBUGONSTART` to `2`, then attach a debugger to the process manually after it starts.
+Set the environment variable `MSBUILDDEBUGONSTART` to control debugger behavior at startup:
+- `1`: Launch debugger for all processes (main MSBuild and TaskHost child processes)
+- `2`: Wait for manual debugger attach for all processes
+- `3`: Launch debugger for main MSBuild process only, skip TaskHost child processes
+
+For example, set `MSBUILDDEBUGONSTART` to `2`, then attach a debugger to the process manually after it starts.
 
 ## Using the repository binaries to perform builds
 

documentation/wiki/Building-Testing-and-Debugging-on-Full-Framework-MSBuild.md

@@ -81,7 +81,10 @@ Sometimes it's useful to patch your copy of Visual Studio in order to test or de
 ### Debugging MSBuild
 
 #### Breakpoints
-To break into the [main method](https://github.com/dotnet/msbuild/blob/bd00d6cba24d41efd6f54699c3fdbefb9f5034a1/src/MSBuild/XMake.cs#L493-L506) of MSBuild.exe: set the environment variable `MSBUILDDEBUGONSTART` to 1 (uses `Debugger.Launch()`) or 2 (waits until debugger is attached).
+To break into the [main method](https://github.com/dotnet/msbuild/blob/bd00d6cba24d41efd6f54699c3fdbefb9f5034a1/src/MSBuild/XMake.cs#L493-L506) of MSBuild.exe: set the environment variable `MSBUILDDEBUGONSTART` to:
+- `1`: Launch debugger for all processes (main MSBuild and TaskHost child processes) using `Debugger.Launch()`
+- `2`: Wait for manual debugger attach for all processes
+- `3`: Launch debugger for main MSBuild process only using `Debugger.Launch()`, skip TaskHost child processes
 
 To break into MSBuild's [BuildManager.BeginBuild](https://github.com/dotnet/msbuild/blob/bd00d6cba24d41efd6f54699c3fdbefb9f5034a1/src/Build/BackEnd/BuildManager/BuildManager.cs#L414) set the environment variable `MSBuildDebugBuildManagerOnStart` to 1 (uses `Debugger.Launch()`) or 2 (waits until debugger is attached).
 This is useful for debugging MSBuild when it is called from other apps that use its APIs instead of its executable (for example Visual Studio). You can also filter which processes trigger the breakpoint by setting `MSBuildDebugProcessName` to a substring of the process name. For example, to trigger the breakpoint only under Visual Studio's top level process you would set `MSBuildDebugProcessName` to the value `devenv`.

src/Build.UnitTests/BackEnd/DebugUtils_tests.cs

@@ -5,6 +5,7 @@
 using System.IO;
 using System.Linq;
 using Microsoft.Build.Shared;
+using Microsoft.Build.Shared.Debugging;
 using Shouldly;
 using Xunit;
 
@@ -38,5 +39,13 @@ public void DumpExceptionToFileShouldWriteInDebugDumpPath()
                 File.Delete(exceptionFile);
             }
         }
+
+        [Fact]
+        public void IsInTaskHostNode_ReturnsFalseForCentralNode()
+        {
+            // When running in the main test process (no /nodemode argument),
+            // we should not be in a TaskHost node
+            DebugUtils.IsInTaskHostNode().ShouldBeFalse();
+        }
     }
 }

src/MSBuild/XMake.cs

@@ -552,6 +552,13 @@ private static void DebuggerLaunchCheck()
                 case "1":
                     Debugger.Launch();
                     break;
+                case "3":
+                    // Value "3" debugs the main MSBuild process but skips debugging child TaskHost processes
+                    if (!DebugUtils.IsInTaskHostNode())
+                    {
+                        Debugger.Launch();
+                    }
+                    break;
 #endif
                 case "2":
                     // Sometimes easier to attach rather than deal with JIT prompt

src/MSBuildTaskHost/OutOfProcTaskHost.cs

@@ -94,6 +94,10 @@ internal static ExitType Execute()
 
                     Console.ReadLine();
                     break;
+                case "3":
+                    // Value "3" skips debugging for TaskHost processes but debugs the main MSBuild process
+                    // This is useful when you want to debug MSBuild but not the child TaskHost processes
+                    break;
             }
 
             bool restart = false;

src/Shared/Debugging/DebugUtils.cs

@@ -105,6 +105,16 @@ private static bool CurrentProcessMatchesDebugName()
 
         public static string DebugPath { get; private set; }
 
+        /// <summary>
+        /// Returns true if the current process is an out-of-proc TaskHost node.
+        /// </summary>
+        /// <returns>
+        /// True if this process was launched with /nodemode:2 (indicating it's a TaskHost process),
+        /// false otherwise. This is useful for conditionally enabling debugging or other behaviors
+        /// based on whether the code is running in the main MSBuild process or a child TaskHost process.
+        /// </returns>
+        public static bool IsInTaskHostNode() => ProcessNodeMode.Value == NodeMode.OutOfProcTaskHostNode;
+
         public static string FindNextAvailableDebugFilePath(string fileName)
         {
             var extension = Path.GetExtension(fileName);