Compile bug: MTT S4000 compile error

[murarduino]

Git commit

cdf94a1

Operating systems

Linux

GGML backends

Musa

Problem description & steps to reproduce

I tried renting a server with 4 MTT S4000 GPUs at AutoDL to test the deployment of llama.cpp. Operating system is Ubuntu 22.04, using the latest clone llama.cpp first confirm that mthreads-gmi can display GPU situation properly, because I believe MTT driver and MUSA SDK should be installed properly

Compile according to MUSA instructions in build.md
cmake -B build -DGGML_MUSA=ON

When I execute build
cmake --build build --config Release
the following error occurs

First Bad Commit

cmake --build build --config Release
[ 3%] Built target ggml-base
[ 3%] Building CXX object ggml/src/ggml-musa/CMakeFiles/ggml-musa.dir/mudnn.cu.o
/root/llama.cpp/ggml/src/ggml-musa/mudnn.cu:25:29: error: no member named 'EXECUTION_FAILED' in 'musa::dnn::Status'
case mudnn::Status::EXECUTION_FAILED:
~~~~~~~~~~~~~~~^
1 error generated when compiling for mp_21.
gmake[2]: *** [ggml/src/ggml-musa/CMakeFiles/ggml-musa.dir/build.make:1168: ggml/src/ggml-musa/CMakeFiles/ggml-musa.dir/mudnn.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1783: ggml/src/ggml-musa/CMakeFiles/ggml-musa.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

Compile command

cmake -B build -DGGML_MUSA=ON
cmake --build build --config Release

Relevant log output

No log file provided

[yeahdongcn]

We've recently switched to MUSA SDK rc4.0.1 (see #13647 for more info), but I noticed that AutoDL is still using an older version (rc3.1.0) for now.

[yeahdongcn]

You might want to reset to an earlier commit for a successful build.
Note: Our previously tested version is MUSA SDK rc3.1.1, which is still different from the one used in the AutoDL instance.

[murarduino]

You might want to reset to an earlier commit for a successful build. Note: Our previously tested version is MUSA SDK rc3.1.1, which is still different from the one used in the AutoDL instance.

Thank you for your response.
I have submitted the issue to AutoDL's customer service. AutoDL may help me upgrade the version of MUSA SDK. In addition, it was also communicated with MTheads that a new version of cloud resources may be used for testing in the later stage.

[whyhhh123]

You might want to reset to an earlier commit for a successful build. Note: Our previously tested version is MUSA SDK rc3.1.1, which is still different from the one used in the AutoDL instance.

Thank you for your response. I have submitted the issue to AutoDL's customer service. AutoDL may help me upgrade the version of MUSA SDK. In addition, it was also communicated with MTheads that a new version of cloud resources may be used for testing in the later stage.

Hello, have you found a way to deploy any large model using S4000 on AutoDl now? I am currently trying to use ollama \ vllm (unable to use Docker internally) and llama.cpp (unable to compile), but have not been successfully deployed.

[murarduino]

You might want to reset to an earlier commit for a successful build. Note: Our previously tested version is MUSA SDK rc3.1.1, which is still different from the one used in the AutoDL instance.

Thank you for your response. I have submitted the issue to AutoDL's customer service. AutoDL may help me upgrade the version of MUSA SDK. In addition, it was also communicated with MTheads that a new version of cloud resources may be used for testing in the later stage.

Hello, have you found a way to deploy any large model using S4000 on AutoDl now? I am currently trying to use ollama \ vllm (unable to use Docker internally) and llama.cpp (unable to compile), but have not been successfully deployed.

No. I have consulted with AutoDL technicians and they have no plans to update MUSA 4.0. I have also consulted with the MThreads engineer, in addition to schedule MThreads product trial, there is no other way.

[whyhhh123]

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。

[yeahdongcn]

It should be straightforward to build this on an AutoDL instance. As mentioned earlier, you can find more details in #13647.

Here’s a quick guide:

git clone https://github.com/ggml-org/llama.cpp.git
git checkout <a commit before 3398305>
cmake -B build -DGGML_MUSA=ON
cmake --build build --config Release

And that’s it!

[whyhhh123]

You might want to reset to an earlier commit for a successful build. Note: Our previously tested version is MUSA SDK rc3.1.1, which is still different from the one used in the AutoDL instance.

Thank you for your response. I have submitted the issue to AutoDL's customer service. AutoDL may help me upgrade the version of MUSA SDK. In addition, it was also communicated with MTheads that a new version of cloud resources may be used for testing in the later stage.

Hello, have you found a way to deploy any large model using S4000 on AutoDl now? I am currently trying to use ollama \ vllm (unable to use Docker internally) and llama.cpp (unable to compile), but have not been successfully deployed.

No. I have consulted with AutoDL technicians and they have no plans to update MUSA 4.0. I have also consulted with the MThreads engineer, in addition to schedule MThreads product trial, there is no other way.

I have successfully deployed the model using S4000 on autodl, and the llama. cpp branch needs to be switched to a branch before 3398305 to compile and run normally