Start building Clang runtimes on-demand (#5338)

This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.

The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.

This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.

It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.

It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.

I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.

I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.

Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.

The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.

---------

Co-authored-by: Jon Ross-Perkins <[email protected]>
Co-authored-by: Richard Smith <[email protected]>

Author: 3 people

.codespell_ignore

@@ -2,6 +2,7 @@
 # Exceptions. See /LICENSE for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
+ArchType
 atleast
 circularly
 compiletime
@@ -14,6 +15,7 @@ forin
 groupt
 indext
 inout
+isELF
 parameteras
 pullrequest
 rightt

MODULE.bazel

@@ -116,6 +116,7 @@ http_archive(
         "@carbon//bazel/llvm_project:0001_Patch_for_mallinfo2_when_using_Bazel_build_system.patch",
         "@carbon//bazel/llvm_project:0002_Added_Bazel_build_for_compiler_rt_fuzzer.patch",
         "@carbon//bazel/llvm_project:0003_Comment_out_unloaded_proto_library_dependencies.patch",
+        "@carbon//bazel/llvm_project:0004_Introduce_filegroups_for_compiler_rt_builtins_runtime.patch",
     ],
     strip_prefix = "llvm-project-{0}".format(llvm_project_version),
     urls = ["https://github.com/llvm/llvm-project/archive/{0}.tar.gz".format(llvm_project_version)],

bazel/llvm_project/0004_Introduce_filegroups_for_compiler_rt_builtins_runtime.patch

@@ -0,0 +1,194 @@
+From 19d5d9913778ca95da272f41c5916907154a5e73 Mon Sep 17 00:00:00 2001
+From: Chandler Carruth <[email protected]>
+Date: Thu, 24 Apr 2025 05:03:43 +0000
+Subject: [PATCH] Introduce filegroups for compiler-rt builtins runtimes
+
+These filegroups allow downstream projects to package and build
+customized runtime libraries.
+
+The filegroups work hard to use globs and a careful structuring to
+create the structured breakdown of sources needed to target different
+architectures and platforms without having to maintain a complete
+parallel list of sources from CMake.
+---
+ .../compiler-rt/BUILD.bazel                   | 167 ++++++++++++++++++
+ 1 file changed, 167 insertions(+)
+
+diff --git a/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel b/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
+index 6a5a89fdee40..7d158f0c13f2 100644
+--- a/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
++++ b/utils/bazel/llvm-project-overlay/compiler-rt/BUILD.bazel
+@@ -128,3 +128,170 @@ cc_library(
+     ],
+     includes = ["lib/fuzzer"],
+ )
++
++BUILTINS_CRTBEGIN_SRCS = ["lib/builtins/crtbegin.c"]
++
++filegroup(
++    name = "builtins_crtbegin_src",
++    srcs = BUILTINS_CRTBEGIN_SRCS,
++)
++
++BUILTINS_CRTEND_SRCS = ["lib/builtins/crtend.c"]
++
++filegroup(
++    name = "builtins_crtend_src",
++    srcs = BUILTINS_CRTEND_SRCS,
++)
++
++# Note that while LLVM's CompilerRT provides a few hosted sources, we don't
++# currently build them:
++#
++# - `emutls.c`: Unclear we need to support targets with software emulated
++#   TLS rather than hardware support.
++# - `enable_execute_stack.c`: Used to implement support for a builtin that
++#   marks part of the stack as *executable* to support the GCC extension of
++#   nested functions. This extension was never implemented in Clang, and is
++#   generally considered a security issue to include. We expect to be able
++#   to avoid even linking the support code for this into binaries at this
++#   point.
++# - `eprintf.c`: This provided a legacy `__eprintf` builtin used by old
++#   versions of `assert.h` in its macros, but does not appear to be needed
++#   when building with modern versions of this header.
++BUILTINS_HOSTED_SRCS = [
++    "lib/builtins/emutls.c",
++    "lib/builtins/enable_execute_stack.c",
++    "lib/builtins/eprintf.c",
++]
++
++filegroup(
++    name = "builtins_hosted_srcs",
++    srcs = BUILTINS_HOSTED_SRCS,
++)
++
++BUILTINS_BF16_SRCS_PATTERNS = [
++    # `bf` marks 16-bit Brain floating-point number builtins.
++    "lib/builtins/*bf*.c",
++]
++
++filegroup(
++    name = "builtins_bf16_srcs",
++    srcs = glob(BUILTINS_BF16_SRCS_PATTERNS),
++)
++
++BUILTINS_X86_FP80_SRCS_PATTERNS = [
++    # `xc` marks 80-bit complex number builtins.
++    "lib/builtins/*xc*.c",
++
++    # `xf` marks 80-bit floating-point builtins.
++    "lib/builtins/*xf*.c",
++]
++
++filegroup(
++    name = "builtins_x86_fp80_srcs",
++    srcs = glob(
++        BUILTINS_X86_FP80_SRCS_PATTERNS,
++        exclude = BUILTINS_BF16_SRCS_PATTERNS,
++    ),
++)
++
++BUILTINS_TF_SRCS_PATTERNS = [
++    # `tc` marks 128-bit complex number builtins.
++    "lib/builtins/*tc*.c",
++
++    # `tf` marks 128-bit floating-point builtins.
++    "lib/builtins/*tf*.c",
++]
++
++BUILTINS_TF_EXCLUDES = (
++    BUILTINS_HOSTED_SRCS +
++    BUILTINS_BF16_SRCS_PATTERNS +
++    BUILTINS_X86_FP80_SRCS_PATTERNS
++)
++
++filegroup(
++    name = "builtins_tf_srcs",
++    srcs = glob(
++        BUILTINS_TF_SRCS_PATTERNS,
++        exclude = BUILTINS_TF_EXCLUDES,
++    ),
++)
++
++BUILTINS_MACOS_ATOMIC_SRCS_PATTERNS = [
++    "lib/builtins/atomic_*.c",
++]
++
++filegroup(
++    name = "builtins_macos_atomic_srcs",
++    srcs = glob(BUILTINS_MACOS_ATOMIC_SRCS_PATTERNS),
++)
++
++filegroup(
++    name = "builtins_aarch64_srcs",
++    srcs = [
++        "lib/builtins/cpu_model/aarch64.c",
++        "lib/builtins/cpu_model/aarch64.h",
++    ] + glob(
++        [
++            "lib/builtins/cpu_model/AArch64*.inc",
++            "lib/builtins/cpu_model/aarch64/**/*.inc",
++            "lib/builtins/aarch64/*.S",
++            "lib/builtins/aarch64/*.c",
++        ],
++        exclude = [
++            # This file isn't intended to directly compile, but to be used to
++            # generate a collection of outline atomic helpers.
++            # TODO: Add support for generating the sources for these helpers if
++            # there are users that need this functionality from the builtins
++            # library.
++            "lib/builtins/aarch64/lse.S",
++        ],
++    ),
++)
++
++filegroup(
++    name = "builtins_x86_arch_srcs",
++    srcs = [
++        "lib/builtins/cpu_model/x86.c",
++        "lib/builtins/i386/fp_mode.c",
++    ],
++)
++
++filegroup(
++    name = "builtins_x86_64_srcs",
++    srcs = glob([
++        "lib/builtins/x86_64/*.c",
++        "lib/builtins/x86_64/*.S",
++    ]),
++)
++
++filegroup(
++    name = "builtins_i386_srcs",
++    srcs = glob(
++        [
++            "lib/builtins/i386/*.c",
++            "lib/builtins/i386/*.S",
++        ],
++        exclude = [
++            # This file is used for both i386 and x86_64.
++            "lib/builtins/i386/fp_mode.c",
++        ],
++    ),
++)
++
++filegroup(
++    name = "builtins_generic_srcs",
++    srcs = ["lib/builtins/cpu_model/cpu_model.h"] + glob(
++        [
++            "lib/builtins/*.c",
++            "lib/builtins/*.h",
++            "lib/builtins/*.inc",
++        ],
++        exclude = (
++            BUILTINS_CRTBEGIN_SRCS +
++            BUILTINS_CRTEND_SRCS +
++            BUILTINS_TF_EXCLUDES +
++            BUILTINS_TF_SRCS_PATTERNS +
++            BUILTINS_MACOS_ATOMIC_SRCS_PATTERNS
++        ),
++    ),
++)
+-- 
+2.49.0.850.g28803427d3-goog
+

scripts/fix_cc_deps.py

@@ -70,7 +70,8 @@ class RuleChoice(NamedTuple):
 IGNORE_SOURCE_FILE_REGEX = re.compile(
     r"^(third_party/clangd.*|common/version.*\.cpp"
     r"|.*_autogen_manifest\.cpp"
-    r"|toolchain/base/llvm_tools.def)$"
+    r"|toolchain/base/llvm_tools.def"
+    r"|toolchain/base/runtime_sources.h)$"
 )
 
 

toolchain/base/BUILD

@@ -4,6 +4,7 @@
 
 load("//bazel/cc_rules:defs.bzl", "cc_library", "cc_test")
 load("llvm_tools.bzl", "LLVM_MAIN_TOOLS", "generate_llvm_tools_def")
+load("runtime_sources.bzl", "generate_runtime_sources_cc_library")
 
 package(default_visibility = ["//visibility:public"])
 
@@ -248,6 +249,8 @@ cc_library(
     ] + [info.lib for info in LLVM_MAIN_TOOLS.values()],
 )
 
+generate_runtime_sources_cc_library(name = "runtime_sources")
+
 cc_library(
     name = "shared_value_stores",
     hdrs = ["shared_value_stores.h"],