Compile koboldcpp with rocWMMA on gfx12

[wasd-tech]

I would like to compile koboldcpp with rocWMMA support on gfx12.
I know that rocWMMA is not compatible with llamacpp and koboldcpp as of ROCm 6.4.1 but by installing rocWMMA from GitHub I can successful compile llamacpp with it and enjoy using flash attention and kv quantization.
I would like to know if it is possible to overwrite the standard behaviour and make koboldcpp compile with rocWMMA on gfx12.
Thanks in advance for the help!

OS:Linux (official Almalinux container for AMD HIP/ROCm)
GPU: Rx 9070 XT

[LostRuins]

I followed the approach in ggml-org#14202

Pull the latest experimental after 18cd46a
then add the flag GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12=1 when building

let me know if this works

[henk717]

For future reference, that also has a rocm version check so once rocm 7 is detected it will become default enabled.

[wasd-tech]

Thanks for the replies!

Pull the latest experimental after 18cd46a then add the flag GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12=1 when building

I will try this on Monday but I think that it will work.

I will also try with the alpha release of ROCm 7.0 on both llamacpp and koboldcpp and see if it already works.

[wasd-tech]

@LostRuins I tried the concedo_experimental branch with the commit that you suggest, but unfortunately I run on these errors:

g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:500: ggml-vec.o] Error 1
make: *** Waiting for unfinished jobs....
cc: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:457: ggml.o] Error 1
make: *** [Makefile:492: ggml-unops.o] Error 1
cc: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:628: ggml_v1.o] Error 1
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:658: expose.o] Error 1
cc: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:596: ggml_v3.o] Error 1
make: *** [Makefile:494: ggml-ops.o] Error 1
cc: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:687: gpttype_adapter.o] Error 1
make: *** [Makefile:477: ggml-cpu.o] Error 1
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:490: ggml-binops.o] Error 1
g++: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:662: sdcpp_default.o] Error 1
cc: error: unrecognized command line option '-GGML_HIP_ROCWMMA_FATTN_GFX12'
make: *** [Makefile:612: ggml_v2.o] Error 1

The command I used is the following:
make LLAMA_HIPBLAS=1 GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12=1 GPU_TARGETS=gfx1201 -j$(nproc)

I used ROCm 6.4.1

[LostRuins]

Apologies, I had a typo in my makefile. Can you pull it try again

[wasd-tech]

I tried again and now it work flawlessly! I can clearly see the huge speed up with flash attention enabled.
Thanks for the help, and keep up the amazing work you do.