[clang][Driver][SPIR-V] Allow linking IR using llvm-link

[sarnex]

SPIR-V does not have a production-grade linker so it is often necessary to stay in LLVM-IR as long as possible and only convert to SPIR-V at the very end.

As such, it is common that we want to create a BC library that is linked into a user program (motivating example here is the OpenMP device RTL). We only convert to SPIR-V at the very end when we have a fully linked program.

Other targets can achieve a similar goal by using LTO to get a linked-BC file, but the SPIR-V linker does not support LTO, so we have no way to do it with clang. We can do it with llvm-link directly, but my understanding is that it is not intended to be directly called in production workflows and does not fit well into code that supports multiple targets.

Extend the existing behavior for HIP that allows -emit-llvm without -c.

[llvmbot]

@llvm/pr-subscribers-clang-driver
@llvm/pr-subscribers-clang

@llvm/pr-subscribers-backend-spir-v

Author: Nick Sarnie (sarnex)

Changes

SPIR-V does not have a production-grade linker so it is often necessary to stay in LLVM-IR as long as possible and only convert to SPIR-V at the very end.

As such, it is common that we want to create a BC library that is linked into a user program (motivating example here is the OpenMP device RTL). We only convert to SPIR-V at the very end when we have a fully linked program.

Other targets can achieve a similar goal by using LTO to get a linked-BC file, but the SPIR-V linker does not support LTO, so we have no way to do it with clang. We can do it with llvm-link directly, but my understanding is that it is not intended to be directly called in production workflows and does not fit well into code that supports multiple targets.

Extend the existing behavior for HIP that allows -emit-llvm without -c.

---

Full diff: https://github.com/llvm/llvm-project/pull/169572.diff

4 Files Affected:

• (modified) clang/lib/Driver/Driver.cpp (+13-3)
• (modified) clang/lib/Driver/ToolChains/SPIRV.cpp (+26)
• (modified) clang/lib/Driver/ToolChains/SPIRV.h (+5)
• (added) clang/test/Driver/spirv-llvm-link.c (+31)

diff --git a/clang/lib/Driver/Driver.cpp b/clang/lib/Driver/Driver.cpp
index de8d4601210ae..27d5bcd4290c6 100644
--- a/clang/lib/Driver/Driver.cpp
+++ b/clang/lib/Driver/Driver.cpp
@@ -4265,8 +4265,11 @@ void Driver::handleArguments(Compilation &C, DerivedArgList &Args,
       Args.AddFlagArg(nullptr,
                       getOpts().getOption(options::OPT_frtlib_add_rpath));
     }
-    // Emitting LLVM while linking disabled except in HIPAMD Toolchain
-    if (Args.hasArg(options::OPT_emit_llvm) && !Args.hasArg(options::OPT_hip_link))
+    // Emitting LLVM while linking disabled except in the HIPAMD or SPIR-V
+    // Toolchains
+    if (Args.hasArg(options::OPT_emit_llvm) &&
+        !Args.hasArg(options::OPT_hip_link) &&
+        !C.getDefaultToolChain().getTriple().isSPIRV())
       Diag(clang::diag::err_drv_emit_llvm_link);
     if (C.getDefaultToolChain().getTriple().isWindowsMSVCEnvironment() &&
         LTOMode != LTOK_None &&
@@ -4595,7 +4598,14 @@ void Driver::BuildDefaultActions(Compilation &C, DerivedArgList &Args,
       LA->propagateHostOffloadInfo(C.getActiveOffloadKinds(),
                                    /*BoundArch=*/nullptr);
     } else {
-      LA = C.MakeAction<LinkJobAction>(LinkerInputs, types::TY_Image);
+      // If we are linking but were passed -emit-llvm, we will be calling
+      // llvm-link, so set the output type accordingly. This is only allowed in
+      // rare cases, so make sure we aren't going to error about it.
+      types::ID LT =
+          Args.hasArg(options::OPT_emit_llvm) && !Diags.hasErrorOccurred()
+              ? types::TY_LLVM_BC
+              : types::TY_Image;
+      LA = C.MakeAction<LinkJobAction>(LinkerInputs, LT);
     }
     if (!UseNewOffloadingDriver)
       LA = OffloadBuilder->processHostLinkAction(LA);
diff --git a/clang/lib/Driver/ToolChains/SPIRV.cpp b/clang/lib/Driver/ToolChains/SPIRV.cpp
index 27de55cfebfc1..ddb2a0bbb5058 100644
--- a/clang/lib/Driver/ToolChains/SPIRV.cpp
+++ b/clang/lib/Driver/ToolChains/SPIRV.cpp
@@ -70,6 +70,28 @@ void SPIRV::constructAssembleCommand(Compilation &C, const Tool &T,
                                          Exec, CmdArgs, Input, Output));
 }
 
+void SPIRV::constructLLVMLinkCommand(Compilation &C, const Tool &T,
+                                     const JobAction &JA,
+                                     const InputInfo &Output,
+                                     const InputInfoList &Inputs,
+                                     const llvm::opt::ArgStringList &Args) {
+  // Construct llvm-link command.
+  // The output from llvm-link is a bitcode file.
+  ArgStringList LlvmLinkArgs;
+
+  assert(!Inputs.empty() && "Must have at least one input.");
+
+  LlvmLinkArgs.append({"-o", Output.getFilename()});
+  for (auto Input : Inputs)
+    LlvmLinkArgs.push_back(Input.getFilename());
+
+  const char *LlvmLink =
+      C.getArgs().MakeArgString(T.getToolChain().GetProgramPath("llvm-link"));
+  C.addCommand(std::make_unique<Command>(JA, T, ResponseFileSupport::None(),
+                                         LlvmLink, LlvmLinkArgs, Inputs,
+                                         Output));
+}
+
 void SPIRV::Translator::ConstructJob(Compilation &C, const JobAction &JA,
                                      const InputInfo &Output,
                                      const InputInfoList &Inputs,
@@ -121,6 +143,10 @@ void SPIRV::Linker::ConstructJob(Compilation &C, const JobAction &JA,
                                  const InputInfoList &Inputs,
                                  const ArgList &Args,
                                  const char *LinkingOutput) const {
+  if (JA.getType() == types::TY_LLVM_BC) {
+    constructLLVMLinkCommand(C, *this, JA, Output, Inputs, {});
+    return;
+  }
   const ToolChain &ToolChain = getToolChain();
   std::string Linker = ToolChain.GetProgramPath(getShortName());
   ArgStringList CmdArgs;
diff --git a/clang/lib/Driver/ToolChains/SPIRV.h b/clang/lib/Driver/ToolChains/SPIRV.h
index 924eb01adcbbf..249053c23b792 100644
--- a/clang/lib/Driver/ToolChains/SPIRV.h
+++ b/clang/lib/Driver/ToolChains/SPIRV.h
@@ -27,6 +27,11 @@ void constructAssembleCommand(Compilation &C, const Tool &T,
                               const InputInfo &Input,
                               const llvm::opt::ArgStringList &Args);
 
+void constructLLVMLinkCommand(Compilation &C, const Tool &T,
+                              const JobAction &JA, const InputInfo &Output,
+                              const InputInfoList &Inputs,
+                              const llvm::opt::ArgStringList &Args);
+
 class LLVM_LIBRARY_VISIBILITY Translator : public Tool {
 public:
   Translator(const ToolChain &TC)
diff --git a/clang/test/Driver/spirv-llvm-link.c b/clang/test/Driver/spirv-llvm-link.c
new file mode 100644
index 0000000000000..9c30654707016
--- /dev/null
+++ b/clang/test/Driver/spirv-llvm-link.c
@@ -0,0 +1,31 @@
+// Check BC input
+// RUN: mkdir -p %t
+// RUN: touch %t/a.bc
+// RUN: touch %t/b.bc
+// RUN: %clang -### --target=spirv64 -emit-llvm %t/a.bc %t/b.bc 2>&1 | FileCheck --check-prefix=CHECK-TOOL-BC %s
+
+// CHECK-TOOL-BC: "-cc1" {{.*}} "-o" "[[TMP1_BC:.+]]" "-x" "ir" "{{.*}}.bc"
+// CHECK-TOOL-BC: "-cc1" {{.*}} "-o" "[[TMP2_BC:.+]]" "-x" "ir" "{{.*}}.bc"
+// CHECK-TOOL-BC: llvm-link{{.*}} "-o" {{.*}} "[[TMP1_BC]]" "[[TMP2_BC]]"
+
+// RUN: %clang -ccc-print-bindings --target=spirv64 -emit-llvm %t/a.bc %t/b.bc 2>&1 | FileCheck -check-prefix=CHECK-BINDINGS-BC %s
+
+// CHECK-BINDINGS-BC: "spirv64" - "clang", inputs: ["{{.*}}.bc"], output: "[[TMP1_BINDINGS_BC:.+]]"
+// CHECK-BINDINGS-BC: "spirv64" - "clang", inputs: ["{{.*}}.bc"], output: "[[TMP2_BINDINGS_BC:.+]]"
+// CHECK-BINDINGS-BC: "spirv64" - "SPIR-V::Linker", inputs: ["[[TMP1_BINDINGS_BC]]", "[[TMP2_BINDINGS_BC]]"], output: "{{.*}}.bc"
+
+// Check source input
+// RUN: touch %t/foo.c
+// RUN: touch %t/bar.c
+
+// RUN: %clang -### --target=spirv64 -emit-llvm %t/foo.c %t/bar.c 2>&1 | FileCheck --check-prefix=CHECK-TOOL-SRC %s
+
+// CHECK-TOOL-SRC: "-cc1" {{.*}} "-o" "[[TMP1_SRC_BC:.+]]" "-x" "c" "{{.*}}foo.c"
+// CHECK-TOOL-SRC: "-cc1" {{.*}} "-o" "[[TMP2_SRC_BC:.+]]" "-x" "c" "{{.*}}bar.c"
+// CHECK-TOOL-SRC: llvm-link{{.*}} "-o" {{.*}} "[[TMP1_SRC_BC]]" "[[TMP2_SRC_BC]]"
+
+// RUN: %clang -ccc-print-bindings --target=spirv64 -emit-llvm %t/foo.c %t/bar.c 2>&1 | FileCheck -check-prefix=CHECK-BINDINGS-SRC %s
+
+// CHECK-BINDINGS-SRC: "spirv64" - "clang", inputs: ["{{.*}}foo.c"], output: "[[TMP1_BINDINGS_SRC_BC:.+]]"
+// CHECK-BINDINGS-SRC: "spirv64" - "clang", inputs: ["{{.*}}bar.c"], output: "[[TMP2_BINDINGS_SRC_BC:.+]]"
+// CHECK-BINDINGS-SRC: "spirv64" - "SPIR-V::Linker", inputs: ["[[TMP1_BINDINGS_SRC_BC]]", "[[TMP2_BINDINGS_SRC_BC]]"], output: "{{.*}}.bc"

[bader]

LLD can link LLVM, do LTO and emit SPIR-V with SPIR-V backend (or LLVM bitcode with -emit-llvm option). Have you considered using LLD for OpenMP device RTL?

[shiltian]

Not sure if that's a good idea to call llvm-link directly. Can't we just call clang?

[sarnex]

That's the problem we're trying to solve, how can we link bitcode with clang directly? The only option I know would be the mlink_bitcode_file group of options, and those are really more for linking a BC file into source code being compiled, rather than to link a group of BC files.
Also, AMD also calls llvm-link directly in their equivalent of this flag --hip-link -emit-llvm.

[sarnex]

LLD can link LLVM, do LTO and emit SPIR-V with SPIR-V backend (or LLVM bitcode with -emit-llvm option). Have you considered using LLD for OpenMP device RTL?

But it can't actually link SPIR-V, so we can't use it when we aren't using LTO, so we would only ever lld when LTO is enabled, which seems somewhat weird, and I think it's more complex than the approach in this PR.

The existing targets for the OpenMP device RTL actually do it this way, by calling lld with -emit-llvm, however to do the same we'd basically have to support LTO in general with SPIR-V in a user-visible way, and that might end up being a decent amount of work.