[Clang][SYCL] Introduce clang-sycl-link-wrapper to link SYCL offloading device code

This PR is one of the many PRs in the SYCL upstreaming effort focusing on
device code linking during the SYCL offload compilation process.
RFC: https://discourse.llvm.org/t/rfc-offloading-design-for-sycl-offload-kind-and-spir-targets/74088

In this PR, we introduce a new tool that will be used to perform device code
linking for SYCL offload kind. It accepts SYCL device objects in LLVM IR bitcode
format and will generate a fully linked device object that can then be wrapped
and linked into the host object.

A primary use case for this tool is to perform device code linking for objects
with SYCL offload kind inside the clang-linker-wrapper.
It can also be invoked via clang driver as follows:

`clang --target=spirv64 --sycl-link input.bc`

Device code linking for SYCL offloading kind has a number of known quirks that
makes it difficult to use in a unified offloading setting.
Two of the primary issues are:
1. Several finalization steps are required to be run on the fully-linked LLVM
IR bitcode to gaurantee conformance to SYCL standards. This step is unique to
SYCL offloading compilation flow.
2. SPIR-V LLVM Translator tool is an extenal tool and hence SPIR-V IR code
generation cannot be done as part of LTO. This limitation will be lifted once
SPIR-V backend is available as a viable LLVM backend.

Hence, we introduce this new tool to provide a clean wrapper to perform SYCL
device linking.

Thanks

Signed-off-by: Arvind Sudarsanam <[email protected]>

Author: asudarsa

clang/docs/ClangSYCLLinkWrapper.rst

@@ -0,0 +1,80 @@
+=======================
+Clang SYCL Link Wrapper
+=======================
+
+.. contents::
+   :local:
+
+.. _clang-sycl-link-wrapper:
+
+Introduction
+============
+
+This tool works as a wrapper around the SYCL device code linking process.
+The purpose of this wrapper is to provide an interface to link SYCL device
+bitcode in LLVM IR format, SYCL device bitcode in SPIR-V IR format, and native
+binary objects, and then use the SPIR-V LLVM Translator tool on fully linked
+device objects to produce the final output.
+After the linking stage, the fully linked device code in LLVM IR format may
+undergo several SYCL-specific finalization steps before the SPIR-V code
+generation step.
+The wrapper will also support the Ahead-Of-Time (AOT) compilation flow. AOT
+compilation is the process of invoking the back-end at compile time to produce
+the final binary, as opposed to just-in-time (JIT) compilation when final code
+generation is deferred until application runtime.
+
+Device code linking for SYCL offloading has several known quirks that
+make it difficult to use in a unified offloading setting. Two of the primary
+issues are:
+1. Several finalization steps are required to be run on the fully linked LLVM
+IR bitcode to guarantee conformance to SYCL standards. This step is unique to
+the SYCL offloading compilation flow.
+2. The SPIR-V LLVM Translator tool is an external tool and hence SPIR-V IR code
+generation cannot be done as part of LTO. This limitation can be lifted once
+the SPIR-V backend is available as a viable LLVM backend.
+
+This tool has been proposed to work around these issues.
+
+Usage
+=====
+
+This tool can be used with the following options. Several of these options will
+be passed down to downstream tools like 'llvm-link', 'llvm-spirv', etc.
+
+.. code-block:: console
+
+  OVERVIEW: A utility that wraps around the SYCL device code linking process.
+  This enables linking and code generation for SPIR-V JIT targets and AOT
+  targets.
+
+  USAGE: clang-sycl-link-wrapper [options]
+
+  OPTIONS:
+    --arch <value>                Specify the name of the target architecture.
+    --dry-run                     Print generated commands without running.
+    -g                            Specify that this was a debug compile.
+    -help-hidden                  Display all available options
+    -help                         Display available options (--help-hidden for more)
+    --library-path=<dir>          Set the library path for SYCL device libraries
+    -o <path>                     Path to file to write output
+    --save-temps                  Save intermediate results
+    --triple <value>              Specify the target triple.
+    --version                     Display the version number and exit
+    -v                            Print verbose information
+    -spirv-dump-device-code=<dir> Directory to dump SPIR-V IR code into
+    -is-windows-msvc-env          Specify if we are compiling under windows environment
+    -llvm-spirv-options=<value>   Pass options to llvm-spirv tool
+
+Example
+=======
+
+This tool is intended to be invoked when targeting any of the target offloading
+toolchains. When the --sycl-link option is passed to the clang driver, the
+driver will invoke the linking job of the target offloading toolchain, which in
+turn will invoke this tool. This tool can be used to create one or more fully
+linked device images that are ready to be wrapped and linked with host code to
+generate the final executable.
+
+.. code-block:: console
+
+  clang-sycl-link-wrapper --triple spirv64 --arch native input.bc

clang/docs/index.rst

@@ -97,6 +97,7 @@ Using Clang Tools
    ClangOffloadBundler
    ClangOffloadPackager
    ClangRepl
+   ClangSYCLLinkWrapper
 
 Design Documents
 ================

clang/include/clang/Driver/Options.td

@@ -6728,7 +6728,10 @@ def fsycl : Flag<["-"], "fsycl">,
 def fno_sycl : Flag<["-"], "fno-sycl">,
   Visibility<[ClangOption, CLOption]>,
   Group<sycl_Group>, HelpText<"Disables SYCL kernels compilation for device">;
-
+def sycl_link : Flag<["--"], "sycl-link">, Flags<[HelpHidden]>,
+  Visibility<[ClangOption, CLOption]>,
+  Group<sycl_Group>, HelpText<"Perform link through clang-sycl-link-wrapper via the target "
+  "offloading toolchain.">;
 // OS-specific options
 let Flags = [TargetSpecific] in {
 defm android_pad_segment : BooleanFFlag<"android-pad-segment">, Group<f_Group>;

clang/lib/Driver/Driver.cpp

@@ -4780,6 +4780,11 @@ Action *Driver::ConstructPhaseAction(
   if (Phase == phases::Assemble && Input->getType() != types::TY_PP_Asm)
     return Input;
 
+  // Use of --sycl-link will only allow for the link phase to occur. This is
+  // for all input files.
+  if (Args.hasArg(options::OPT_sycl_link) && Phase != phases::Link)
+    return Input;
+
   // Build the appropriate action.
   switch (Phase) {
   case phases::Link:

clang/lib/Driver/ToolChains/SPIRV.cpp

@@ -95,7 +95,19 @@ void SPIRV::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
 
+  // Use of --sycl-link will call the clang-sycl-link-wrapper instead of
+  // the default linker (spirv-link).
+  if (Args.hasArg(options::OPT_sycl_link))
+    Linker = ToolChain.GetProgramPath("clang-sycl-link-wrapper");
   C.addCommand(std::make_unique<Command>(JA, *this, ResponseFileSupport::None(),
                                          Args.MakeArgString(Linker), CmdArgs,
                                          Inputs, Output));
 }
+
+SPIRVToolChain::SPIRVToolChain(const Driver &D, const llvm::Triple &Triple,
+                               const ArgList &Args)
+    : ToolChain(D, Triple, Args) {
+  NativeLLVMSupport = Args.hasArg(options::OPT_sycl_link);
+}
+
+bool SPIRVToolChain::HasNativeLLVMSupport() const { return NativeLLVMSupport; }

clang/lib/Driver/ToolChains/SPIRV.h

@@ -57,8 +57,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
 
 public:
   SPIRVToolChain(const Driver &D, const llvm::Triple &Triple,
-                 const llvm::opt::ArgList &Args)
-      : ToolChain(D, Triple, Args) {}
+                 const llvm::opt::ArgList &Args);
 
   bool useIntegratedAs() const override { return true; }
 
@@ -72,6 +71,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
   }
   bool isPICDefaultForced() const override { return false; }
   bool SupportsProfiling() const override { return false; }
+  bool HasNativeLLVMSupport() const override;
 
   clang::driver::Tool *SelectTool(const JobAction &JA) const override;
 
@@ -81,6 +81,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
 
 private:
   clang::driver::Tool *getTranslator() const;
+  bool NativeLLVMSupport;
 };
 
 } // namespace toolchains

clang/test/Driver/Inputs/libsycl-complex.bc

@@ -0,0 +1,9 @@
+// Tests the clang-sycl-link-wrapper tool
+//
+// Test a simple case without arguments
+// RUN: %clangxx -fsycl -emit-llvm -c %s -o %t.bc
+// RUN: clang-sycl-link-wrapper --dry-run -triple spirv64 %t.bc --library-path=%S/Inputs -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=CMDS
+// CMDS: "{{.*}}llvm-link{{.*}}" {{.*}}.bc -o [[FIRSTLLVMLINKOUT:.*]].bc --suppress-warnings
+// CMDS-NEXT: "{{.*}}llvm-link{{.*}}" -only-needed [[FIRSTLLVMLINKOUT]].bc {{.*}}libsycl-crt.bc {{.*}}libsycl-complex.bc -o [[SECONDLLVMLINKOUT:.*]].bc --suppress-warnings
+// CMDS-NEXT: "{{.*}}llvm-spirv{{.*}}" {{.*}}-o a.spv [[SECONDLLVMLINKOUT]].bc

clang/test/Driver/Inputs/libsycl-crt.bc

@@ -0,0 +1,7 @@
+// Tests the driver when linking LLVM IR bitcode files and targeting SPIR-V
+// architecture.
+//
+// RUN: touch %t.bc
+// RUN: %clangxx --target=spirv64 --sycl-link -### %t.bc 2>&1 \
+// RUN:   | FileCheck %s -check-prefix=LINK
+// LINK: "{{.*}}clang-sycl-link-wrapper{{.*}}" "{{.*}}.bc" "-o" "a.out"