Merge pull request #19691 from kouvel/Tier22_4

This is a port of several changes that went into master after 2.2 forked, including dependencies for, and enabling tiered compilation by default in 2.2. Quick summary of commits is below, see the commit descriptions and PRs for more info.
- Commit 1 - Fix nested spin locks in thread pool etw firing (#17677)
  - Fixes a lock nesting issue when there is an ETW listener, which can occur without tiering, but is almost deterministic with tiering enabled because the first event that is fired typically hits this code path
- Commit 2 - Don't close the JIT func info file on shutdown (#18060)
  - Fixes a crash during shutdown that only occurs when JIT logging is enabled (typically in the coreclr tests and CI). More frequent with tiering enabled because of different JIT timing and background jitting.
- Commit 3 - Apply tiering's call counting delay more broadly (#18610)
  - Fixes a perf issue when tiering is enabled in server first-request scenarios where there is a significant gap between process startup and first request
- Commit 4 - Changes only affect debug builds - Eliminate arm64 contract asserts (#19015)
  - Fixes some incorrect asserts that trigger more frequently with tiering
- Commit 5 - Use 16 bytes to spill SIMD12 (#19237)
  - Fixes a crash in corefx System.Numerics.Tests.Vector3Tests.Vector3EqualsTest. Occurs with minopt JIT or with tiering.
- Commit 6 - Fix an apartment state issue (partial port of #19384)
  - This is a partial port of this PR (only the portion that addresses issue #17822)
  - This is a breaking change, though a minor one that we have concluded is an acceptable risk to take for 2.2
  - Fixes a behavioral difference that can be seen more easily tiering enabled in APIs on the `Thread` class relevant to apartment state. The issue can also be seen in some cases when tiering is disabled.
- Commit 7 - Enable Tiered Compilation by default (#19525)
  - Enables tiering by default, can be disabled through environment, or through .csproj/.json when using dotnet
  - Removes deprecated config variable (EXPERIMENTAL_TieredCompilation) that was previously exposed in 2.1 along with the current config variable (TieredCompilation), along with miscellaneous test fixes
- Commit 8 - Changes only affect tests - Fix tiered compilation option for case-sensitive systems (#19567)
  - Fixes tiering environment variable casing for non-Windows platforms
- Commit 9 - Disable tiered compilation on arm64
  - There is an open issue that may be partly related to minopts on arm64 (https://github.com/dotnet/coreclr/issues/18895). Disabling tiering by default on arm64 to limit exposing new issues.

This change would be followed up with dotnet/corefx#31822
- Adds tests for Commit 6 - Fix an apartment state issue (partial port of #19384)
  - Changes only affect tests

Closes https://github.com/dotnet/coreclr/issues/18973

Author: kouvel

Documentation/project-docs/clr-configuration-knobs.md

@@ -770,8 +770,7 @@ Name | Description | Type | Class | Default Value | Flags
 
 Name | Description | Type | Class | Default Value | Flags 
 -----|-------------|------|-------|---------------|-------
-`EXPERIMENTAL_TieredCompilation` | Deprecated - Use COMPLUS_TieredCompilation | `DWORD` | `UNSUPPORTED` | `0` | 
-`TieredCompilation` | Enables tiered compilation | `DWORD` | `EXTERNAL` | `0` | 
+`TieredCompilation` | Enables tiered compilation | `DWORD` | `EXTERNAL` | `1` | 
 `TieredCompilation_Test_CallCounting` | Enabled by default (only activates when TieredCompilation is also enabled). If disabled immediately backpatches prestub, and likely prevents any tier1 promotion | `DWORD` | `UNSUPPORTED` | `1` | 
 `TieredCompilation_Test_OptimizeTier0` | Use optimized codegen (normally used by tier1) in tier0 | `DWORD` | `UNSUPPORTED` | `0` | 
 `TieredCompilation_Tier1CallCountingDelayMs` | Delay in milliseconds since process startup or the last tier 0 JIT before call counting begins for tier 1 promotion. | `DWORD` | `UNSUPPORTED` | `100` | 

src/inc/CrstTypes.def

@@ -787,3 +787,7 @@ End
 Crst ReadyToRunEntryPointToMethodDescMap
     AcquiredBefore ExecuteManRangeLock
 End
+
+Crst TieredCompilation
+    AcquiredBefore ThreadpoolTimerQueue
+End

src/inc/clrconfigvalues.h

@@ -650,10 +650,14 @@ RETAIL_CONFIG_DWORD_INFO(INTERNAL_HillClimbing_GainExponent,
 /// Tiered Compilation
 ///
 #ifdef FEATURE_TIERED_COMPILATION
+#ifdef _TARGET_ARM64_
 RETAIL_CONFIG_DWORD_INFO(EXTERNAL_TieredCompilation, W("TieredCompilation"), 0, "Enables tiered compilation")
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_LEGACY_TieredCompilation, W("EXPERIMENTAL_TieredCompilation"), 0, "Deprecated - Use COMPLUS_TieredCompilation")
+#else // !_TARGET_ARM64_
+RETAIL_CONFIG_DWORD_INFO(EXTERNAL_TieredCompilation, W("TieredCompilation"), 1, "Enables tiered compilation")
+#endif // _TARGET_ARM64_
 RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Tier1CallCountThreshold, W("TieredCompilation_Tier1CallCountThreshold"), 30, "Number of times a method must be called after which it is promoted to tier 1.")
-RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Tier1CallCountingDelayMs, W("TieredCompilation_Tier1CallCountingDelayMs"), 100, "Delay in milliseconds since process startup or the last tier 0 JIT before call counting begins for tier 1 promotion.")
+RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Tier1CallCountingDelayMs, W("TieredCompilation_Tier1CallCountingDelayMs"), 100, "A perpetual delay in milliseconds that is applied to tier 1 call counting and jitting, while there is tier 0 activity.")
+RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Tier1DelaySingleProcMultiplier, W("TieredCompilation_Tier1DelaySingleProcMultiplier"), 10, "Multiplier for TieredCompilation_Tier1CallCountingDelayMs that is applied on a single-processor machine or when the process is affinitized to a single processor.")
 
 RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Test_CallCounting, W("TieredCompilation_Test_CallCounting"), 1, "Enabled by default (only activates when TieredCompilation is also enabled). If disabled immediately backpatches prestub, and likely prevents any tier1 promotion")
 RETAIL_CONFIG_DWORD_INFO(UNSUPPORTED_TieredCompilation_Test_OptimizeTier0, W("TieredCompilation_Test_OptimizeTier0"), 0, "Use optimized codegen (normally used by tier1) in tier0")

src/inc/crsttypes.h

@@ -176,18 +176,19 @@ enum CrstType
     CrstThreadpoolWorker = 157,
     CrstThreadStaticDataHashTable = 158,
     CrstThreadStore = 159,
-    CrstTPMethodTable = 160,
-    CrstTypeEquivalenceMap = 161,
-    CrstTypeIDMap = 162,
-    CrstUMEntryThunkCache = 163,
-    CrstUMThunkHash = 164,
-    CrstUniqueStack = 165,
-    CrstUnresolvedClassLock = 166,
-    CrstUnwindInfoTableLock = 167,
-    CrstVSDIndirectionCellLock = 168,
-    CrstWinRTFactoryCache = 169,
-    CrstWrapperTemplate = 170,
-    kNumberOfCrstTypes = 171
+    CrstTieredCompilation = 160,
+    CrstTPMethodTable = 161,
+    CrstTypeEquivalenceMap = 162,
+    CrstTypeIDMap = 163,
+    CrstUMEntryThunkCache = 164,
+    CrstUMThunkHash = 165,
+    CrstUniqueStack = 166,
+    CrstUnresolvedClassLock = 167,
+    CrstUnwindInfoTableLock = 168,
+    CrstVSDIndirectionCellLock = 169,
+    CrstWinRTFactoryCache = 170,
+    CrstWrapperTemplate = 171,
+    kNumberOfCrstTypes = 172
 };
 
 #endif // __CRST_TYPES_INCLUDED
@@ -358,6 +359,7 @@ int g_rgCrstLevelMap[] =
     11,			// CrstThreadpoolWorker
     4,			// CrstThreadStaticDataHashTable
     10,			// CrstThreadStore
+    9,			// CrstTieredCompilation
     9,			// CrstTPMethodTable
     3,			// CrstTypeEquivalenceMap
     7,			// CrstTypeIDMap
@@ -534,6 +536,7 @@ LPCSTR g_rgCrstNameMap[] =
     "CrstThreadpoolWorker",
     "CrstThreadStaticDataHashTable",
     "CrstThreadStore",
+    "CrstTieredCompilation",
     "CrstTPMethodTable",
     "CrstTypeEquivalenceMap",
     "CrstTypeIDMap",

src/inc/utilcode.h

@@ -1431,6 +1431,7 @@ class CPUGroupInfo
     static BOOL m_threadUseAllCpuGroups;
     static WORD m_initialGroup;
     static CPU_Group_Info *m_CPUGroupInfoArray;
+    static bool s_hadSingleProcessorAtStartup;
 
     static BOOL InitCPUGroupInfoAPI();
     static BOOL InitCPUGroupInfoArray();
@@ -1485,6 +1486,13 @@ class CPUGroupInfo
     static void ChooseCPUGroupAffinity(GROUP_AFFINITY *gf);
     static void ClearCPUGroupAffinity(GROUP_AFFINITY *gf);
 #endif
+
+public:
+    static bool HadSingleProcessorAtStartup()
+    {
+        LIMITED_METHOD_CONTRACT;
+        return s_hadSingleProcessorAtStartup;
+    }
 };
 
 int GetCurrentProcessCpuCount();

src/jit/compiler.cpp

@@ -1437,14 +1437,6 @@ void Compiler::compShutdown()
     }
 #endif // FEATURE_JIT_METHOD_PERF
 
-#if FUNC_INFO_LOGGING
-    if (compJitFuncInfoFile != nullptr)
-    {
-        fclose(compJitFuncInfoFile);
-        compJitFuncInfoFile = nullptr;
-    }
-#endif // FUNC_INFO_LOGGING
-
 #if COUNT_RANGECHECKS
     if (optRangeChkAll > 0)
     {

src/jit/compiler.h

@@ -708,6 +708,7 @@ class LclVarDsc
         // For 32-bit architectures, we make local variable SIMD12 types 16 bytes instead of just 12. We can't do
         // this for arguments, which must be passed according the defined ABI. We don't want to do this for
         // dependently promoted struct fields, but we don't know that here. See lvaMapSimd12ToSimd16().
+        // (Note that for 64-bits, we are already rounding up to 16.)
         if ((lvType == TYP_SIMD12) && !lvIsParam)
         {
             assert(lvExactSize == 12);

src/jit/regset.cpp

@@ -3274,10 +3274,11 @@ var_types Compiler::tmpNormalizeType(var_types type)
 
     type = genActualType(type);
 
-#if defined(FEATURE_SIMD) && !defined(_TARGET_64BIT_)
-    // For SIMD on 32-bit platforms, we always spill SIMD12 to a 16-byte SIMD16 temp.
-    // This is because we don't have a single instruction to store 12 bytes. We also
-    // allocate non-argument locals as 16 bytes; see lvSize().
+#if defined(FEATURE_SIMD)
+    // We always spill SIMD12 to a 16-byte SIMD16 temp.
+    // This is because we don't have a single instruction to store 12 bytes, so we want
+    // to ensure that we always have the full 16 bytes for loading & storing the value.
+    // We also allocate non-argument locals as 16 bytes; see lvSize().
     if (type == TYP_SIMD12)
     {
         type = TYP_SIMD16;

src/utilcode/util.cpp

@@ -852,13 +852,14 @@ BYTE * ClrVirtualAllocWithinRange(const BYTE *pMinAddr,
 }
 #endif
 
-/*static*/ BOOL  CPUGroupInfo::m_enableGCCPUGroups = FALSE;
-/*static*/ BOOL  CPUGroupInfo::m_threadUseAllCpuGroups = FALSE;
-/*static*/ WORD  CPUGroupInfo::m_nGroups = 0;
-/*static*/ WORD  CPUGroupInfo::m_nProcessors = 0;
-/*static*/ WORD  CPUGroupInfo::m_initialGroup = 0;
+/*static*/ BOOL CPUGroupInfo::m_enableGCCPUGroups = FALSE;
+/*static*/ BOOL CPUGroupInfo::m_threadUseAllCpuGroups = FALSE;
+/*static*/ WORD CPUGroupInfo::m_nGroups = 0;
+/*static*/ WORD CPUGroupInfo::m_nProcessors = 0;
+/*static*/ WORD CPUGroupInfo::m_initialGroup = 0;
 /*static*/ CPU_Group_Info *CPUGroupInfo::m_CPUGroupInfoArray = NULL;
-/*static*/ LONG   CPUGroupInfo::m_initialization = 0;
+/*static*/ LONG CPUGroupInfo::m_initialization = 0;
+/*static*/ bool CPUGroupInfo::s_hadSingleProcessorAtStartup = false;
 
 // Check and setup function pointers for >64 LP Support
 /*static*/ BOOL CPUGroupInfo::InitCPUGroupInfoAPI()
@@ -1066,6 +1067,18 @@ DWORD LCM(DWORD u, DWORD v)
 	m_enableGCCPUGroups = enableGCCPUGroups && hasMultipleGroups;
 	m_threadUseAllCpuGroups = threadUseAllCpuGroups && hasMultipleGroups;
 #endif // _TARGET_AMD64_ || _TARGET_ARM64_
+
+    // Determine if the process is affinitized to a single processor (or if the system has a single processor)
+    DWORD_PTR processAffinityMask, systemAffinityMask;
+    if (GetProcessAffinityMask(GetCurrentProcess(), &processAffinityMask, &systemAffinityMask))
+    {
+        processAffinityMask &= systemAffinityMask;
+        if (processAffinityMask != 0 && // only one CPU group is involved
+            (processAffinityMask & (processAffinityMask - 1)) == 0) // only one bit is set
+        {
+            s_hadSingleProcessorAtStartup = true;
+        }
+    }
 }
 
 /*static*/ BOOL CPUGroupInfo::IsInitialized()

src/vm/appdomain.cpp

@@ -3078,8 +3078,9 @@ void SystemDomain::SetThreadAptState (Thread::ApartmentState state)
         Thread::ApartmentState pState = pThread->SetApartment(Thread::AS_InSTA, TRUE);
         _ASSERTE(pState == Thread::AS_InSTA);
     }
-    else if (state == Thread::AS_InMTA)
+    else
     {
+        // If an apartment state was not explicitly requested, default to MTA
         Thread::ApartmentState pState = pThread->SetApartment(Thread::AS_InMTA, TRUE);
         _ASSERTE(pState == Thread::AS_InMTA);
     }