Add support for experimental targets

[rva3]

Allow building experimental targets (such as ARC, CSKY, M68k, Xtensa)

Tested with ./build-llvm.py --bolt --lto thin -L ../../linux-next --pgo kernel-defconfig-slim -t AArch64 ARM X86 Mips -T CSKY

[msfjarvis]

Looks like you'll have to reformat the file with yapf before CI passes.

[rva3]

..yeah. Sorry for inconvenience. Force pushed again.

[msfjarvis]

LGTM, I'll let Nathan make the final call.

[nathanchance]

I won’t be able to have a closer look at this until Monday but it seems to me like we should be able to make this work without a separate --experimental-targets (i.e. just make the user provide both regular and experimental targets under --targets then do the split up under the hood). I think that would be a little better so that people’s same command lines will continue to work regardless of the status of the backend. Additionally, this would make it easier to share the code between get_all_targets() and get_experimental_targets() with an experimental parameter.

[nathanchance]

Something like this is what I would have in mind as a diff on top of this change:

nathanchance@906c66b

As a diff against main:

diff --git a/tc_build/llvm.py b/tc_build/llvm.py
index f2214c1..aa6e044 100644
--- a/tc_build/llvm.py
+++ b/tc_build/llvm.py
@@ -15,11 +15,19 @@ import tc_build.utils
 LLVM_VER_FOR_RUNTIMES = 20
 
 
-def get_all_targets(llvm_folder):
+def get_all_targets(llvm_folder, experimental=False):
+    variables = ['LLVM_ALL_TARGETS']
+    if experimental:
+        variables.append('LLVM_ALL_EXPERIMENTAL_TARGETS')
+    targets = []
+
     contents = Path(llvm_folder, 'llvm/CMakeLists.txt').read_text(encoding='utf-8')
-    if not (match := re.search(r'set\(LLVM_ALL_TARGETS([\w|\s]+)\)', contents)):
-        raise RuntimeError('Could not find LLVM_ALL_TARGETS?')
-    return [val for target in match.group(1).splitlines() if (val := target.strip())]
+    for variable in variables:
+        if not (match := re.search(fr"set\({variable}([\w|\s]+)\)", contents)):
+            raise RuntimeError(f"Could not find {variables}?")
+        targets += [val for target in match.group(1).splitlines() if (val := target.strip())]
+
+    return targets
 
 
 class LLVMBuilder(Builder):
@@ -305,10 +313,24 @@ class LLVMBuilder(Builder):
             self.cmake_defines['LLVM_ENABLE_WARNINGS'] = 'OFF'
         if self.tools.llvm_tblgen:
             self.cmake_defines['LLVM_TABLEGEN'] = self.tools.llvm_tblgen
-        self.cmake_defines['LLVM_TARGETS_TO_BUILD'] = ';'.join(self.targets)
         if self.tools.ld:
             self.cmake_defines['LLVM_USE_LINKER'] = self.tools.ld
 
+        standard_targets = get_all_targets(self.folders.source)
+        llvm_targets_to_build = []
+        llvm_experimental_targets_to_build = []
+        for target in self.targets:
+            # We know that the targets are valid from the validate_targets()
+            # call above, meaning that if they are not in the standard targets
+            # list, they must be experimental targets.
+            if target in standard_targets:
+                llvm_targets_to_build.append(target)
+            else:
+                llvm_experimental_targets_to_build.append(target)
+        self.cmake_defines['LLVM_TARGETS_TO_BUILD'] = ';'.join(llvm_targets_to_build)
+        self.cmake_defines['LLVM_EXPERIMENTAL_TARGETS_TO_BUILD'] = ';'.join(
+            llvm_experimental_targets_to_build)
+
         # Clear Linux needs a different target to find all of the C++ header files, otherwise
         # stage 2+ compiles will fail without this
         # We figure this out based on the existence of x86_64-generic-linux in the C++ headers path
@@ -404,7 +426,7 @@ class LLVMBuilder(Builder):
         if not self.targets:
             raise RuntimeError('No targets set?')
 
-        all_targets = get_all_targets(self.folders.source)
+        all_targets = get_all_targets(self.folders.source, experimental=True)
 
         for target in self.targets:
             if target in ('all', 'host'):
@@ -414,7 +436,7 @@ class LLVMBuilder(Builder):
                 # tuple() for shorter pretty printing versus instead of
                 # ('{"', '".join(all_targets)}')
                 raise RuntimeError(
-                    f"Requested target ('{target}') was not found in LLVM_ALL_TARGETS {tuple(all_targets)}, check spelling?"
+                    f"Requested target ('{target}') was not found in LLVM_ALL_TARGETS or LLVM_ALL_EXPERIMENTAL_TARGETS {tuple(all_targets)}, check spelling?"
                 )
 
 

[nathanchance]

I have put up #314 as an alternative to this pull request. I did a light test but it would be good if @rva3 could verify it works as well as this PR did.

[rva3]

#314 works fine for me too. I guess this one should be closed then.