[libclc] Add an option to build SPIR-V targets with the LLVM backend #151347
Conversation
This removes the dependency on an external tool to build the SPIR-V files. It may be of interest to projects such as Mesa. Note that the option is off by default as using the SPIR-V backend, at least on my machine, uses a *lot* of memory and the process is often killed in a parallelized build. It does complete, however. Fixes llvm#135327.
I see the same behavior, goes up to something like 30GB. The input is not particularly large (just 300k lines of IR). From a quick run through massif, it looks like MachineInstrs in SPIRVGlobalRegistry are the memory hog? |
| ) | ||
| if ( LIBCLC_USE_SPIRV_BACKEND ) | ||
| add_custom_command( OUTPUT ${libclc_builtins_lib} | ||
| COMMAND ${clang_exe} --target=${ARG_TRIPLE} -x ir -o ${libclc_builtins_lib} ${builtins_link_lib} |
There was a problem hiding this comment.
should -c be added to clang command line? It avoids spirv-link step.
There was a problem hiding this comment.
Good question! I'm not sure, to be honest. I tried using -c and it generates a slightly larger file (6900 vs 6792). It seems to take a few fewer seconds to do so, which is an upside. I haven't looked into what exactly it's doing.
There was a problem hiding this comment.
perhaps spirv-link does optimization, but obviously there is only one file to link. LGTM as the final size is smaller, so this is not meant to be a blocker.
|
Can this be backported to 21? |
|
I don't think it makes sense to backport this. At least at the time it landed it only worked in the sense of "does not crash during compilation". I'm not sure if the big compile-time and memory usage issue has been resolved since, or anyone has checked whether the produced SPIRV actually works. |
|
Yeah, it looks like at least 638383c is needed on 21 to make it work Just that infact |
|
Hi. When trying to build Here we begin compiling a bc file: 
Still compiling the same file, memory goes up to 62GB + 8GB of swap: 
Here's the failing result: 
The file being compiled at the time of high memory consumption is this: > ls -hla /builddir/build/BUILD/llvm-22.0.0_pre20251106.g93ef57617d080f-build/llvm-project-93ef57617d080f6eac7a064e09f3e4145c334ae8/llvm/redhat-linux-build/tools/libclc/obj.libclc.dir/amdgcn--amdhsa/builtins.opt.amdgcn--amdhsa.bc
-rw-r--r--. 1 mockbuild mock 6.6M Nov 25 10:16 /builddir/build/BUILD/llvm-22.0.0_pre20251106.g93ef57617d080f-build/llvm-project-93ef57617d080f6eac7a064e09f3e4145c334ae8/llvm/redhat-linux-build/tools/libclc/obj.libclc.dir/amdgcn--amdhsa/builtins.opt.amdgcn--amdhsa.bcThe code I test with is from Nov. 6th 2025 93ef576. I try to build with a newer revision of LLVM but I think it is worth mentioning this. |
|
Okay, I ran a test again with revision from 25th November 2025 (2f8e712). Here's how I can reproduce it: bash-5.3# cd /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/tools/libclc
bash-5.3# /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/bin/clang-22 --target=spirv64-- -x ir -o /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/./bin/../lib/clang/22/lib/libclc/spirv64-mesa3d-.spv /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/tools/libclc/obj.libclc.dir/spirv64-mesa3d-/builtins.link.spirv64-mesa3d-.bc
clang-22: error: unable to execute command: posix_spawn failed: No such file or directory
clang-22: error: spirv-link command failed with exit code 1 (use -v to see invocation)Here're the bc file details: bash-5.3# ls -lha /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/tools/libclc/obj.libclc.dir/spirv64-mesa3d-/builtins.link.spirv64-mesa3d-.bc
-rw-r--r--. 1 mockbuild mock 2.3M Nov 25 10:52 /builddir/build/BUILD/llvm-22.0.0_pre20251125.g2f8e71287542a5-build/llvm-project-2f8e71287542a597be246d34699c93345d096f22/llvm/redhat-linux-build/tools/libclc/obj.libclc.dir/spirv64-mesa3d-/builtins.link.spirv64-mesa3d-.bcBut this time the memory consumption only jumped to 34GB and no swap. This is probably due to Here's a screen shot right before the error (this time with 
And here's one screenshot after the error: 
|
|
Hi! Yeah I saw the same thing when I was playing around with this (see the PR description). At the time I remember attributing it to the GlobalISel used by SPIR-V. I didn't have any further data points to see whether it was an issue specific to the SPIR-V backend or generally a GlobalISel problem. SPIR-V does custom things with book-keeping which might account for the memory usage. I'm no longer working on this project, but I'd suggest seeing if other GlobalISel backends see a similar degradation in performance. If not, it's probably worth filing a ticket with the SPIR-V folks. I don't have any good suggestions about how to work with what we've got. We could create smaller modules, or reduce parallelisation for the SPIR-V targets? Anything done would in all likelihood be a brittle workaround to the build system. |
|
Thank you @frasercrmck for the feedback. I see what I can do. For now I just want to add two massif profiles that I created: Here's one visualized: 
@nikic, unfortunately I couldn't verify this with my abilities. Both profiles seem to have no notion of $ ms_print massif.out.15 | grep SPIRVGlobalRegistry
$ ms_print massif.out.17 | grep SPIRVGlobalRegistryUpdate: When running valgrind with the default |
5 participants
This removes the dependency on an external tool to build the SPIR-V files. It may be of interest to projects such as Mesa.
Note that the option is off by default as using the SPIR-V backend, at least on my machine, uses a lot of memory and the process is often killed in a parallelized build. It does complete, however.
Fixes #135327.