Fix hardware exception context extraction on Rosetta (#107199)

The recently added AVX support in hardware exception handling path on macOS x64
has introduced a problem when running under Rosetta.
When we extract the floating point part of the context of the failing thread,
the thread can have AVX or AVX512 active, or none of these. The code accidentally
leaves CONTEXT_XSTATE set on the context even when no AVX was enabled on the thread.

Rosetta doesn't support AVX, so having CONTEXT_XSTATE set in the context flags
can lead to later call to RtlRestoreContext attempting to set ymm registers using
instructions that Rosetta cannot emulate and the app crashes due to that.

This doesn't happen in .NET 9, since we always clear the CONTEXT_XSTATE before
exception handling stack unwinding. But in .NET 8, this causes stack overflow
under Rosetta, since the attemt to execute the ymm instruction triggers the
hardware exception handling again and again.

Co-authored-by: Jan Vorlicek <[email protected]>

Author: github-actions[bot]

src/coreclr/pal/src/thread/context.cpp

@@ -1496,6 +1496,12 @@ CONTEXT_GetThreadContextFromThreadState(
 
                 // AMD64's FLOATING_POINT includes the xmm registers.
                 memcpy(&lpContext->Xmm0, &pState->__fpu_xmm0, 16 * 16);
+
+                if (threadStateFlavor == x86_FLOAT_STATE64)
+                {
+                     // There was just a floating point state, so make sure the CONTEXT_XSTATE is not set
+                     lpContext->ContextFlags &= ~(CONTEXT_XSTATE & CONTEXT_AREA_MASK);
+                }
             }
             break;
         }