[clang] Pass fp128 indirectly and return in xmm0 on Windows (#115052)

Clang currently passes and returns `__float128` in vector registers on
MinGW targets, which is LLVM's default ABI for `fp128`. However, the
Windows x86-64 calling convention [1] states the following:

    __m128 types, arrays, and strings are never passed by immediate
    value. Instead, a pointer is passed to memory allocated by the
    caller. Structs and unions of size 8, 16, 32, or 64 bits, and __m64
    types, are passed as if they were integers of the same size. Structs
    or unions of other sizes are passed as a pointer to memory allocated
    by the caller. For these aggregate types passed as a pointer,
    including __m128, the caller-allocated temporary memory must be
    16-byte aligned.

Based on the above it sounds like `__float128` should be passed
indirectly. Thus, change `f128` passing to use the stack and make the
return in xmm0 explicit. This is the identical to `i128`, and passing is
the same as GCC.

Regarding return values, the documentation states:

    A scalar return value that can fit into 64 bits, including the __m64
    type, is returned through RAX. Non-scalar types including floats,
    doubles, and vector types such as __m128, __m128i, __m128d are
    returned in XMM0.

This makes it sound like it should be acceptable to return `__float128`
in xmm0; however, GCC returns `__float128` on the stack. That above ABI
statement as well as consistency with `i128` (which is returned in xmm0)
mean that it would likely be better for GCC to change its return ABI to
match Clang rather than the other way around, so that portion is left
as-is.

Clang's MSVC targets do not support `__float128` or `_Float128`, but
these changes would also apply there if it is eventually enabled.

With [2] which should land around the same time, LLVM will also
implement this ABI so it is not technically necessary for Clang to make
a change here as well. This is sill done in order to be consistent with
other types, and to allow calling convention-aware optimizations at all
available optimization layers (@rnk mentioned possible reuse of stack
arguments). An added benefit is readibility of the LLVM IR since it more
accurately reflects what the lowered assembly does.

[1]:
https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170
[2]: #128848

Author: tgross35

clang/lib/CodeGen/Targets/X86.cpp

@@ -3390,6 +3390,9 @@ ABIArgInfo WinX86_64ABIInfo::classify(QualType Ty, unsigned &FreeSSERegs,
 
     case BuiltinType::Int128:
     case BuiltinType::UInt128:
+    case BuiltinType::Float128:
+      // 128-bit float and integer types share the same ABI.
+
       // If it's a parameter type, the normal ABI rule is that arguments larger
       // than 8 bytes are passed indirectly. GCC follows it. We follow it too,
       // even though it isn't particularly efficient.
@@ -3400,6 +3403,8 @@ ABIArgInfo WinX86_64ABIInfo::classify(QualType Ty, unsigned &FreeSSERegs,
 
       // Mingw64 GCC returns i128 in XMM0. Coerce to v2i64 to handle that.
       // Clang matches them for compatibility.
+      // NOTE: GCC actually returns f128 indirectly but will hopefully change.
+      // See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115054#c8.
       return ABIArgInfo::getDirect(llvm::FixedVectorType::get(
           llvm::Type::getInt64Ty(getVMContext()), 2));
 

clang/test/CodeGen/win-fp128.c

@@ -0,0 +1,19 @@
+// RUN: %clang_cc1 -triple x86_64-windows-gnu -emit-llvm -o - %s \
+// RUN:    | FileCheck %s --check-prefix=CHECK-GNU64
+// __float128 is unsupported on MSVC
+
+__float128 fp128_ret(void) { return 0; }
+// CHECK-GNU64: define dso_local <2 x i64>  @fp128_ret()
+
+__float128 fp128_args(__float128 a, __float128 b) { return a * b; }
+// CHECK-GNU64: define dso_local <2 x i64> @fp128_args(ptr noundef %0, ptr noundef %1)
+
+void fp128_vararg(int a, ...) {
+  // CHECK-GNU64-LABEL: define dso_local void @fp128_vararg
+  __builtin_va_list ap;
+  __builtin_va_start(ap, a);
+  __float128 i = __builtin_va_arg(ap, __float128);
+  // CHECK-GNU64: load ptr, ptr
+  // CHECK-GNU64: load fp128, ptr
+  __builtin_va_end(ap);
+}