Fixed cooperative matrix shaders on Vulkan builds when CROSS_COMPILE ON #2914

sandrohanea · 2025-03-20T18:10:26Z

Cross-Compilation System for GGML Vulkan

Key Components and Flow

glslc Executable Handling:

The system now finds the glslc executable on the host system first (before loading Vulkan) using find_program(Vulkan_GLSLC_EXECUTABLE glslc NO_CMAKE_FIND_ROOT_PATH)
This ensures that we use a host version of glslc even in cross-compilation scenarios
Users can override this by setting the Vulkan_GLSLC_EXECUTABLE cache variable directly (-DVulkan_GLSLC_EXECUTABLE=/path/to/glslc)

Cooperative Matrix Support Detection:

In normal builds, the system tests if the host glslc supports cooperative matrices by compiling test shaders
In cross-compilation, it makes intelligent guesses based on the target platform:
- Android targets default to COOPMAT=ON and may enable COOPMAT2 for newer API/NDK versions
- ARM64 platforms default to COOPMAT=ON but keep COOPMAT2=OFF
- Other platforms default to both OFF
These defaults can be overridden with explicit CMake options:
- -DGGML_VULKAN_COOPMAT_GLSLC_SUPPORT=ON
- -DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT=ON

Shader Compilation Process:

During cross-compilation, the build system creates a host-native version of the vulkan-shaders-gen tool
This tool receives the glslc path from the main CMake and embeds it as a default at compile-time
The tool can still accept a runtime override via the --glslc command-line parameter
This approach ensures that shader compilation always uses a compatible host glslc

Android NDK Compatibility:

The system now properly handles the case where different Android NDKs might have different glslc versions
Users can explicitly select which glslc to use, allowing them to pick a newer version if needed
This solves potential compatibility issues with older NDK glslc versions

CI (in Whisper.net)

As whisper.cpp doesn't have some CI for Vulkan, I'm attaching my CI runs for whisper.net (only building, no tests yet :( ):

With Cross-Compile:
https://github.com/sandrohanea/whisper.net/actions/runs/14092352637/job/39471912246

Without Cross-Compile:
https://github.com/sandrohanea/whisper.net/actions/runs/14092371745/job/39471978181

sandrohanea · 2025-03-26T13:27:51Z

Hello @ggerganov , @slaren ,

Can one of you, please, take a look at this one when you have some time?

Or any other approach to pass the GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT when cross-compiling would be highly appreciated.

slaren · 2025-03-26T13:40:39Z

@0cc4m @jeffbolznv @netrunnereve Can you take a look at this?

jeffbolznv

I'm supportive of this change, just have a couple questions since I'm not a cmake expert.

jeffbolznv · 2025-03-26T14:12:48Z

ggml/src/ggml-vulkan/CMakeLists.txt

                             ../../include/ggml-vulkan.h
                            )
+    if (CMAKE_CROSSCOMPILING)
+        # Make this configurable for cross builds


I don't understand this part. Why does cross compiling skip the checks?

Before this change, it was not skipping this check, but it was not forwarding it to the vulkan-shaders-gen as for CMAKE_CROSSCOMPILING it is starting it with ExternalProject_Add so the compile definition added with add_compile_definitions(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT) was not forwarded to that external project (only added to the current project.

For not CMAKE_CROSSCOMPILING it was added with add_subdirectory which maintained the compile definitions:

After my change, for CMAKE_CROSSCOMPILING, we don't try to detect the version of the coopmap based on the current system but retrieving it from the options and pass it to the external project as well.

Does it work if you remove this condition? i.e. detect whether it's supported and pass it to the external project?

Yes, that should work:

if we remove this condition

detect in the main project if we have the COOPMAT and which version

pass it to the vulkan-shaders-gen external project

in the CMakeLists for that one, we should still check the option and call target_compile_definitions

The reason why I didn't use this approach was that I was thinking that it is clearer to not check the build host for this version when CROSS_COMPILING as we're not building for this system => but indeed, if this system won't have that specific version, we won't be able to build anyway.

I'm happy to simplify it like that if you think it would be better :)

Link to CI in Whisper.net after the change: https://github.com/sandrohanea/whisper.net/actions/runs/14087208133/job/39454313299

jeffbolznv · 2025-03-26T14:14:04Z

ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt

+
 set(TARGET vulkan-shaders-gen)
 add_executable(${TARGET} vulkan-shaders-gen.cpp)
+if (GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)


In ggml-org/llama.cpp#11695 (comment) I had guessed that we may need to pass through a cmake variable telling this makefile which version of glslc to use? Is that not necessary? It's just the #defines that weren't making it through?

I'm not 100% sure if we need additional version to be used (and in my case it was working as same version as the host system was used). I can confirm that only providing the compile definitions is working for my build => https://github.com/sandrohanea/whisper.net/actions/runs/13977409227/job/39134498314

…SUPPORT and DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT based on detected values

jeffbolznv · 2025-03-26T15:31:33Z

ggml/src/ggml-vulkan/CMakeLists.txt

                    ERROR_VARIABLE glslc_error)

    if (${glslc_error} MATCHES ".*extension not supported: GL_NV_cooperative_matrix2.*")
        message(STATUS "GL_NV_cooperative_matrix2 not supported by glslc")


Should there be an "OFF" here like there is for coopmat1 above?

Good catch!
I got too fast on it, thanks for catching it!

It would not be consistent otherwise => won't cause issues as default will be OFF anyway, but I think it is better to have it explicit.

sandrohanea · 2025-03-26T15:40:30Z

CI in Whisper.net for:

https://github.com/sandrohanea/whisper.net/actions/runs/14087392764/job/39454949037 => CMAKE_CROSSCOMPILE
https://github.com/sandrohanea/whisper.net/actions/runs/14087392764/job/39454949037 => NOT CMAKE_CROSSCOMPILE

jeffbolznv

The change looks good to me.

I'm still concerned about a case where the cross-compile picks up a different glslc in the host environment vs the target environment, and if those glslcs don't have the same level of support then we may still have the same problem. I think this would be most likely to happen for android builds, where the android NDK glslc is very old. If that happens, we could fix it by also passing the glslc executable through cmake variables. I'd also be OK with you doing that now, if you want. It should be OK to use a newer glslc, since the code it generates is backward compatible.

0cc4m · 2025-03-26T16:37:17Z

@bandoti is the most knowledgeable about cmake and cross compiling, I think.

… specification for cross-compilation

sandrohanea · 2025-03-26T17:12:03Z

The change looks good to me.

I'm still concerned about a case where the cross-compile picks up a different glslc in the host environment vs the target environment, and if those glslcs don't have the same level of support then we may still have the same problem. I think this would be most likely to happen for android builds, where the android NDK glslc is very old. If that happens, we could fix it by also passing the glslc executable through cmake variables. I'd also be OK with you doing that now, if you want. It should be OK to use a newer glslc, since the code it generates is backward compatible.

Implemented this option to allow the args for different glslc :
3275375

bandoti · 2025-03-26T17:13:23Z

There are a couple of PRs related to this in the Llama codebase:

Fix for the GGML_VULKAN_COOPMAT_GLSLC_SUPPORT:
ggml-org/llama.cpp#12272

And I am working on adding cross-compile to CI here (also Llama):
ggml-org/llama.cpp#12428

@sandrohanea Here is the Github CI build that does not require the Vulkan SDK on Linux, which should work with Ubuntu 24. This is possible by installing glslc and, in this case, libvulkan-dev:riscv64. What I am seeing at the moment is the same error message, but I am not yet convinced this is due to mismatched glslc as there is only one installed in this case—still need to test/verify that however.

CC: @Icenowy

bandoti · 2025-03-26T17:44:48Z

ggml/src/ggml-vulkan/CMakeLists.txt

 find_package(Vulkan COMPONENTS glslc REQUIRED)

+# Allow explicitly specifying glslc executable for cross-compilation scenarios
+if(DEFINED GGML_VULKAN_GLSLC_EXECUTABLE)


I am not convinced that providing a command-line override is the right solution here. When cross-compiling the standard behavior is to separate build/target utilities via the find-root path. For example, the line above is stating that it requires glslc, but it seems this should not in fact be required but found separately using a appropriate paths.

For example, what we are basically saying is:

We need to find Vulkan as defined in the cross-compile toolchain.

We need the build system glslc no matter what.

As such, it would be more consistent with general cross-compiling to instead use find_program(VULKAN_GLSLC_EXECUTABLE glslc REQUIRED NO_CMAKE_FIND_ROOT_PATH) and change the Vulkan find command to find_package(Vulkan) (not specifying the glslc component there).

Thinking about the Android case though, where there are multiple NDKs, and each might have its own glslc might be worthwhile to support both modes. So, the default behavior could be to use find_program, but also allow the command-line option to ensure we have our bases covered. There have been a couple issues surrounding this so probably best to play it safe.

Okay, so I just took a quick look in the FindVulkan.cmake file on my system and quickly checked the docs. Based on the docs, the search for glslc will not be repeated if it is cached first:

This command is used to find a program. A cache entry, or a normal variable if NO_CACHE is specified, named by is created to store the result of this command. If the program is found the result is stored in the variable and the search will not be repeated unless the variable is cleared. If nothing is found, the result will be -NOTFOUND.

One possibility is to set the cache variable Vulkan_GLSLC_EXECUTABLE before calling find_package(Vulkan COMPONENTS glslc REQUIRED). So, first running find_program(Vulkan_GLSLC_EXECUTABLE glslc REQUIRED NO_CMAKE_FIND_ROOT_PATH) would override the search. Similarly, the command-line approach overriding the Vulkan_GLSLC_EXECUTABLE cache variable would override that, and then we don't need a separate variable for the same thing. Something to consider. 🙂

Thinking about the Android case though, where there are multiple NDKs, and each might have its own glslc might be worthwhile to support both modes.

IMO we should just always use the glslc installed on the build system. The NDK glslc is egregiously out of date, it's really not what we want to be using.

Updated the description of the PR, let me know what do you think

…andling and enhance cross-compilation support Detect what is possible but allow outside configuration (Vulkan_GLSLC_EXECUTABLE, GGML_VULKAN_COOPMAT_GLSLC_SUPPORT, GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)

bandoti

After looking over this a bit more (see review notes) I think we should consider the simplicity of llama.cpp #12272, which directly addresses the need to fix only the coopmat shader generation. I think it is in our best interest to resolve that PR first (since it came first anyhow) and circle back to this one for handling the user-specified glslc.

Note: I applied a quick patch to the cross-build CI PR I have been working on to see if it can fix the issues. Had some sync issues with the CI image (unrelated to the change) so I might have to wait a bit before re-testing, but I will post a status update here once it completes.

cc: @Icenowy

bandoti · 2025-03-27T13:41:25Z

ggml/src/ggml-vulkan/CMakeLists.txt

                             ../../include/ggml-vulkan.h
                            )
+    if (CMAKE_CROSSCOMPILING)
+        # Check if user has explicitly set these options


I am not sure what is the need for these Android checks? Perhaps I confused matters mentioning Android, but I believe it was working correctly with the feature detection—unless I am missing something. Feature detection by attempting to compile the shaders should be able to work on all systems.

bandoti · 2025-03-27T13:44:19Z

ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt

+    set(GLSLC_EXECUTABLE ${Vulkan_GLSLC_EXECUTABLE})
+    message(STATUS "Using glslc from parent CMake: ${GLSLC_EXECUTABLE}")
+    # Also define this at compile time to set the default value in the shader generator
+    add_compile_definitions(VULKAN_GLSLC_EXECUTABLE="${GLSLC_EXECUTABLE}")


Actually, I don't think we need this functionality at all. The vulkan-shaders-gen binary has a command-line switch which receives a path to the specified glslc. There is really no need to provide a statically compiled path to glslc, since it is already being passed in at runtime.

bandoti · 2025-03-27T13:46:18Z

ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp

+// Default glslc path, can be overridden at compile time or runtime 
+#ifdef VULKAN_GLSLC_EXECUTABLE
+    // If VULKAN_GLSLC_EXECUTABLE is defined at compile time, use it
+    #define DEFAULT_GLSLC_EXECUTABLE VULKAN_GLSLC_EXECUTABLE


See note above. I don't think there's any benefit to bake an executable path into the binary when it is already supplied as a command-line switch to vulkan-shaders-gen.

sandrohanea · 2025-03-28T18:38:25Z

I see that ggml-org/llama.cpp#12272 was merged 🚀
That should be enough for the main issue of the CROSS_COMPILE. I was trying to allow various versions of glslc as suggested in the #2914 (review) but not sure if that is needed. Definetly not for my case in whisper.net (where only linux build is cross-compiled, not Android, for now).

Will close this PR and if different glslc versions are needed, we can create another one as a follow-up.

Thank you all for your valuable inputs!

bandoti · 2025-03-28T18:41:23Z

Sounds good—please feel free to loop me in if that scenario comes up! I'm happy to help.

Fixed COOPMAT on Vulkan builds with CROSS_COMPILE ON

04ab3d3

sandrohanea mentioned this pull request Mar 20, 2025

Preparing 1.8.0 sandrohanea/whisper.net#363

Merged

Fixed a warning to retrigger CI (failed transiently last time)

e6ab39f

jeffbolznv reviewed Mar 26, 2025

View reviewed changes

Simplified the condition to just pass the DGGML_VULKAN_COOPMAT_GLSLC_…

62e5be8

…SUPPORT and DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT based on detected values

jeffbolznv reviewed Mar 26, 2025

View reviewed changes

Add status messages for GL_NV_cooperative_matrix2 support in Vulkan

eaa2808

jeffbolznv approved these changes Mar 26, 2025

View reviewed changes

Enhance Vulkan CMake configuration to allow explicit glslc executable…

3275375

… specification for cross-compilation

bandoti reviewed Mar 26, 2025

View reviewed changes

WIP Refactor Vulkan CMake configuration to improve glslc executable h…

3519d2d

…andling and enhance cross-compilation support Detect what is possible but allow outside configuration (Vulkan_GLSLC_EXECUTABLE, GGML_VULKAN_COOPMAT_GLSLC_SUPPORT, GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)

jeffbolznv mentioned this pull request Mar 27, 2025

vulkan: fix coopmat shader generation when cross-compiling ggml-org/llama.cpp#12272

Merged

bandoti reviewed Mar 27, 2025

View reviewed changes

sandrohanea closed this Mar 28, 2025

sandrohanea deleted the vulkan-investigation branch March 28, 2025 18:39

sandrohanea mentioned this pull request Mar 30, 2025

[Vulkan] StackOverflow when running ggml-vulkan on NVidia GPU on Windows #2965

Closed

Fixed cooperative matrix shaders on Vulkan builds when CROSS_COMPILE ON #2914

Fixed cooperative matrix shaders on Vulkan builds when CROSS_COMPILE ON #2914

Uh oh!

Conversation

sandrohanea commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cross-Compilation System for GGML Vulkan

Key Components and Flow

glslc Executable Handling:

Cooperative Matrix Support Detection:

Shader Compilation Process:

Android NDK Compatibility:

CI (in Whisper.net)

Uh oh!

sandrohanea commented Mar 26, 2025

Uh oh!

slaren commented Mar 26, 2025

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandrohanea commented Mar 26, 2025

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

0cc4m commented Mar 26, 2025

Uh oh!

sandrohanea commented Mar 26, 2025

Uh oh!

bandoti commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bandoti left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandrohanea commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bandoti commented Mar 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

sandrohanea commented Mar 20, 2025 •

edited

Loading

bandoti commented Mar 26, 2025 •

edited

Loading

bandoti left a comment •

edited

Loading

sandrohanea commented Mar 28, 2025 •

edited

Loading