Issues related to Environment.ProcessorCount calculation

[AntonLapounov]

Below is how we calculate the processor count (Environment.ProcessorCount) on Windows at present:

• A. If DOTNET_PROCESSOR_COUNT setting is defined and its value is in the [1,0xffff] range, use that value and skip all the steps below. This setting overrides everything.
• B1. If the process is 64-bit, both DOTNET_GCCpuGroup and DOTNET_Thread_UseAllCpuGroups are set, and the machine has more than one processor group, use the number of active processors across all processor groups — call GetLogicalProcessorInformationEx(RelationGroup) and sum ActiveProcessorCount fields for all returned groups. Go to step C.
• B2. Otherwise, call GetProcessAffinityMask and use the number of bits set in the process affinity mask if it is not zero. Go to step C.
• B3. If the returned process affinity mask is zero, which may happen if the process already has threads in multiple processor groups, use 64. Go to step C.
• C. If the process is associated with a Windows job, and the job has the CPU limit enabled, use that limit to cap the value obtained in steps B1–B3.

Here is the list of issues with the current approach:

I1. Starting with Windows 11 and Windows Server 2022, on a system with more than 64 processors, process and thread affinities span all processors in the system, across all processor groups, by default. The GetProcessAffinityMask function returns the process affinity mask for the calling thread's primary group (which by default is the same as the process' primary group). That means step B2 may return too small values. For instance, on an 80-core machine with two processor groups (64+16 processors), step B2 returns 16 for a half of 64-bit processes and 64 for the other half, while all 80 processors are available to each process.

I2. For 32-bit processes GetProcessAffinityMask caps the returned process affinity mask to 32 bits, which may cause step B2 to return too small values. For instance, on a 64-core machine with a single processor group, step B2 may return 32, while all 64 processors are available.

I3. For 32-bit processes querying processor topology in step B1 is never performed, DOTNET_GCCpuGroup and DOTNET_Thread_UseAllCpuGroups settings are ignored. That means we never use the correct value on machines with several processor groups.

Noteworthy, we used to have correct processor group-aware calculation of the processor count for 32-bit processes, which was removed during refactoring and reused in MSBuild sources as a workaround.

My thoughts on this:

• We should fix behavior of 64-bit processes on Windows 11. We may use RtlGetVersion to get Windows version to avoid compatibility shimming since the new behavior is enabled even if you set the executable's compatibility to one of previous Windows versions. We may use GetProcessGroupAffinity to get the list of processor groups available to the process and calculate the sum of ActiveProcessorCount fields for them. My assumption is that it is not possible to impose both processor group affinity and processor affinity within a group at the same time.

• We may want to fix calculation of the processor count for 32-bit processes by restoring the logic that used GetLogicalProcessorInformationEx(RelationGroup) using GetLogicalProcessorInformationEx(RelationProcessorCore), which is not subject to capping.

Note that we have an old work item to enable using all processor groups by default on all supported Windows versions, which has been postponed several times.

EDIT: Clarified that 32-bit processes should use GetLogicalProcessorInformationEx(RelationProcessorCore) to get the correct processor count.

I also suspect that case B3 (GetProcessAffinityMask returning the zero process affinity mask) is not possible in Windows 11 according to the documentation.

[AntonLapounov]

@janvorli @jkotas @kouvel @Maoni0 FYI

[jkotas]

We should fix behavior of 64-bit processes on Windows 11

Agree.

Should we also stop artificially distributing threads across CPU groups and just let Win 11+ OSes deal with it? https://github.com/dotnet/runtime/search?q=ChooseThreadCPUGroupAffinity

My assumption is that it is not possible to impose both processor group affinity and processor affinity within a group at the same time.

Double-check this assumption. It is not obvious to me that this should be impossible.

We may want to fix calculation of the processor count for 32-bit processes by restoring the logic that used GetLogicalProcessorInformationEx(RelationGroup)

Does the 64-bit CPU-group based algorithm is going to produce the correct results on 32-bit platforms? If yes, we can just use that instead of having a special path for 32-bit that uses GetLogicalProcessorInformationEx.

DOTNET_GCCpuGroup and DOTNET_Thread_UseAllCpuGroups

Should we completely ignore these settings on Win11+ OSes? We may also want change the default on pre-Win11 OSes so that the .NET runtime behavior is consistent between OS versions (ie finally fix #13465).

[AntonLapounov]

Does the 64-bit CPU-group based algorithm is going to produce the correct results on 32-bit platforms?

It would be if we use GetLogicalProcessorInformationEx(RelationProcessorCore) instead of GetLogicalProcessorInformationEx(RelationGroup). I have clarified in the top post that our old managed code used the latter (so it was not correct after all). MSBuild first copied our code but then changed it to use the former.

Should we completely ignore these settings on Win11+ OSes? We may also want change the default on pre-Win11 OSes so that the .NET runtime behavior is consistent between OS versions (ie finally fix #13465).

That would be a good plan. There are not many places in our code where we use those settings and those places have to be properly reviewed.

[kouvel]

Should we also stop artificially distributing threads across CPU groups and just let Win 11+ OSes deal with it?

Last I saw, there was an issue with that when using server GC. When threads in the process are affinitized to processors from all CPU groups (GC threads), the scheduler seems to revert to round-robin scheduling of other threads in the process, which kind of defeats the benefit. I think it would be a good thing to do if GC threads were not to be affinitized to specific processors, or to look into a fix to the scheduler.

We may also want change the default on pre-Win11 OSes so that the .NET runtime behavior is consistent between OS versions (ie finally fix #13465).

The behavior would be similar between Win10 and Win11 when these settings are enabled. The main difference would be for threads created from outside the runtime and enter the runtime, the runtime doesn't do any thread-spreading between groups for them, so they by default on Win10 would be confined to the primary CPU group for the process and on Win11 would not. I don't think it's a significant difference from the runtime's perspective, there has already been a case where the runtime's thread-spreading was disabled through config for the process to better balance CPU usage between threads created by the runtime and threads created from outside.

[AntonLapounov]

Last I saw, there was an issue with that when using server GC.

Are you describing the scenario with the Thread_AssignCpuGroups switch disabled?

[kouvel]

Yes, IIRC it was tested on a 64-proc AMD machine with multiple numa nodes at low load and it was seen that CPU time is spread between all numa nodes equally. When the process was affinitized to one numa node, under the same load it used about 50% CPU in that node and performance was much better.

[kouvel]

The same was seen on a 2-socket Intel processor with 32 procs each, though the perf didn't degrade as much when not affinitized. With workstation GC, the issue disappeared and threads were being scheduled as expected, using one numa node / socket first.

[AntonLapounov]

Below are suggestions I received from the OS team. Steps 1–3 are pretty much what I suggested above. At present CLR disrespects affinity restrictions set on the job object (step 4, QueryInformationJobObject(NULL, JobObjectGroupInformationEx)) and restrictions imposed via CPU set API (GetSystemCpuSetInformation). They also confirmed that 32-bit processes cannot properly figure out the number of processors available to them.

I don’t think that there is a good single API today that will capture all the various restrictions on the process to determine the set of processors that are actually available to it.
That is definitely a gap and we have work items filed to develop a better solution.

Right now to get an accurate estimate you’d need to combine information from a number of APIs (CPU sets for example are maintained separately from affinity so to handle a CPU set restriction we need to query it separately).

I think the best estimate that you can get today of the process affinity would be:

1. Call GetProcessGroupAffinity to determine which groups the process has access to
2. If the result is a single group then GetProcessAffinityMask will return the process affinity
3. Otherwise, assume that the affinity has not been restricted at the process level and encompasses all processors in those groups
   a. GetLogicalProcessorInformationEx(RelationGroup) provides us this initial estimate
4. Call QueryInformationJobObject(JobObjectGroupInformationEx) to determine if any affinity restriction has been applied at the job object level
   a. AND the resulting affinities with the process affinity we estimated above

Assuming no one (either inside your process or outside) is fiddling with thread affinities that should give you the full process affinity.

For the process allowed CPU set this can be done by calling GetSystemCpuSetInformation and taking all processors that are either:

1. Not allocated, or
2. Allocated but also AllocatedToTargetProcess

If the intersection of the allowed CPU set and the process affinity is non-empty, then that is the set of processors available to you.
If the intersection is empty, then the allowed CPU set restriction is ignored and the process affinity gives you the set of processors available to you.

That should handle the common management mechanisms for restricting your process.

It doesn’t handle the process default being changed or individual thread affinitization, but that would generally be done by the process itself after creation.
If that’s a concern you could additionally query the ProcessDefaultCpuSets.

It also doesn’t handle systems where heterogeneous processor policies may impact you (e.g. on Surface Pro X low importance work is restricted to only running on the little cores and such policies are likely to be seen on other heterogenous systems).
We don’t have any APIs that would give insight into those power policies today.