-
Notifications
You must be signed in to change notification settings - Fork 13.4k
ggml: riscv: add riscv spacemit backend #15288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ggerganov
merged 19 commits into
ggml-org:master
from
spacemit-com:add-spacemit-backend
Sep 29, 2025
+4,501
−0
Merged
Changes from 17 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
6f8d979
ggml: add spacemit backend
co-seven 0f8d88f
add new line at end of file
alex-spacemit 2ad4450
add riscv zba extension limit
alex-spacemit df1e680
fixed for Merge commit '62c3b645c56ad5288aa401901f043b050edac5a2' int…
alex-spacemit c991354
fixed for review comments, file renamed and format
alex-spacemit 6daf027
Merge commit '69ffd891631befa9e6b485fd646a16dab4f2c007' into add-spac…
alex-spacemit dbc5bd6
fixed for code format, after clang-format
alex-spacemit bd62ed2
use _Float16 instead of __fp16
alex-spacemit 986d8bd
add ci for riscv64-spacemit-ime-native
alex-spacemit 230b258
Merge commit '264f1b51872c125e23fa0ac1da5e2a1170de9a08' into add-spac…
alex-spacemit dba48d8
update debian-13-riscv64-spacemit-ime-native ci label
alex-spacemit 8dcbfc6
remove license comment for spacemit ime
alex-spacemit aea33c0
upgrade binutils for gcc ime
alex-spacemit 4e7ab9e
add spacemit ime cross jobs
alex-spacemit 3e6ba84
remove native compile for riscv64-spacemit-ime
alex-spacemit 46fe5a1
ci : add caching for spacemit ime cross toolchain
alex-spacemit 33ced1f
ci: bug fixed for cache path and env
alex-spacemit 40c8d7e
Update .github/workflows/build-linux-cross.yml for cache path
alex-spacemit b3eec83
bugfixed for build-linux-cross.yml, syntax error
alex-spacemit File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
set(CMAKE_SYSTEM_NAME Linux) | ||
set(CMAKE_SYSTEM_PROCESSOR riscv64) | ||
set(CMAKE_SYSTEM_VERSION 1) | ||
|
||
if (CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "^(riscv)") | ||
message(STATUS "HOST SYSTEM ${CMAKE_HOST_SYSTEM_PROCESSOR}") | ||
else() | ||
set(GNU_MACHINE riscv64-unknown-linux-gnu CACHE STRING "GNU compiler triple") | ||
if (DEFINED ENV{RISCV_ROOT_PATH}) | ||
file(TO_CMAKE_PATH $ENV{RISCV_ROOT_PATH} RISCV_ROOT_PATH) | ||
else() | ||
message(FATAL_ERROR "RISCV_ROOT_PATH env must be defined") | ||
endif() | ||
|
||
set(RISCV_ROOT_PATH ${RISCV_ROOT_PATH} CACHE STRING "root path to riscv toolchain") | ||
set(CMAKE_C_COMPILER ${RISCV_ROOT_PATH}/bin/riscv64-unknown-linux-gnu-gcc) | ||
set(CMAKE_CXX_COMPILER ${RISCV_ROOT_PATH}/bin/riscv64-unknown-linux-gnu-g++) | ||
set(CMAKE_STRIP ${RISCV_ROOT_PATH}/bin/riscv64-unknown-linux-gnu-strip) | ||
set(CMAKE_FIND_ROOT_PATH "${RISCV_ROOT_PATH}/riscv64-unknown-linux-gnu") | ||
set(CMAKE_SYSROOT "${RISCV_ROOT_PATH}/sysroot") | ||
endif() | ||
|
||
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) | ||
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) | ||
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) | ||
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) | ||
set(CMAKE_C_FLAGS "-march=rv64gcv_zfh_zba_zicbop -mabi=lp64d ${CMAKE_C_FLAGS}") | ||
set(CMAKE_CXX_FLAGS "-march=rv64gcv_zfh_zba_zicbop -mabi=lp64d ${CXX_FLAGS}") | ||
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -latomic") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
> [!IMPORTANT] | ||
> This build documentation is specific only to RISC-V SpacemiT SOCs. | ||
|
||
## Build llama.cpp locally (for riscv64) | ||
|
||
1. Prepare Toolchain For RISCV | ||
~~~ | ||
wget https://archive.spacemit.com/toolchain/spacemit-toolchain-linux-glibc-x86_64-v1.1.2.tar.xz | ||
~~~ | ||
|
||
2. Build | ||
Below is the build script: it requires utilizing RISC-V vector instructions for acceleration. Ensure the `GGML_CPU_RISCV64_SPACEMIT` compilation option is enabled. The currently supported optimization version is `RISCV64_SPACEMIT_IME1`, corresponding to the `RISCV64_SPACEMIT_IME_SPEC` compilation option. Compiler configurations are defined in the `riscv64-spacemit-linux-gnu-gcc.cmake` file. Please ensure you have installed the RISC-V compiler and set the environment variable via `export RISCV_ROOT_PATH={your_compiler_path}`. | ||
```bash | ||
|
||
cmake -B build \ | ||
-DCMAKE_BUILD_TYPE=Release \ | ||
-DGGML_CPU_RISCV64_SPACEMIT=ON \ | ||
-DLLAMA_CURL=OFF \ | ||
-DGGML_RVV=ON \ | ||
-DGGML_RV_ZFH=ON \ | ||
-DGGML_RV_ZICBOP=ON \ | ||
-DRISCV64_SPACEMIT_IME_SPEC=RISCV64_SPACEMIT_IME1 \ | ||
-DCMAKE_TOOLCHAIN_FILE=${PWD}/cmake/riscv64-spacemit-linux-gnu-gcc.cmake \ | ||
-DCMAKE_INSTALL_PREFIX=build/installed | ||
|
||
cmake --build build --parallel $(nproc) --config Release | ||
|
||
pushd build | ||
make install | ||
popd | ||
``` | ||
|
||
## Simulation | ||
You can use QEMU to perform emulation on non-RISC-V architectures. | ||
|
||
1. Download QEMU | ||
~~~ | ||
wget https://archive.spacemit.com/spacemit-ai/qemu/jdsk-qemu-v0.0.14.tar.gz | ||
~~~ | ||
|
||
2. Run Simulation | ||
After build your llama.cpp, you can run the executable file via QEMU for simulation, for example: | ||
~~~ | ||
export QEMU_ROOT_PATH={your QEMU file path} | ||
export RISCV_ROOT_PATH_IME1={your RISC-V compiler path} | ||
|
||
${QEMU_ROOT_PATH}/bin/qemu-riscv64 -L ${RISCV_ROOT_PATH_IME1}/sysroot -cpu max,vlen=256,elen=64,vext_spec=v1.0 ${PWD}/build/bin/llama-cli -m ${PWD}/models/Qwen2.5-0.5B-Instruct-Q4_0.gguf -t 1 | ||
~~~ | ||
## Performance | ||
#### Quantization Support For Matrix | ||
~~~ | ||
model name : Spacemit(R) X60 | ||
isa : rv64imafdcv_zicbom_zicboz_zicntr_zicond_zicsr_zifencei_zihintpause_zihpm_zfh_zfhmin_zca_zcd_zba_zbb_zbc_zbs_zkt_zve32f_zve32x_zve64d_zve64f_zve64x_zvfh_zvfhmin_zvkt_sscofpmf_sstc_svinval_svnapot_svpbmt | ||
mmu : sv39 | ||
uarch : spacemit,x60 | ||
mvendorid : 0x710 | ||
marchid : 0x8000000058000001 | ||
~~~ | ||
|
||
Q4_0 | ||
| Model | Size | Params | backend | threads | test | t/s | | ||
| -----------| -------- | ------ | ------- | ------- | ---- |------| | ||
Qwen2.5 0.5B |403.20 MiB|630.17 M| cpu | 4 | pp512|64.12 ± 0.26| | ||
Qwen2.5 0.5B |403.20 MiB|630.17 M| cpu | 4 | tg128|10.03 ± 0.01| | ||
Qwen2.5 1.5B |1011.16 MiB| 1.78 B | cpu | 4 | pp512|24.16 ± 0.02| | ||
Qwen2.5 1.5B |1011.16 MiB| 1.78 B | cpu | 4 | tg128|3.83 ± 0.06| | ||
Qwen2.5 3B | 1.86 GiB | 3.40 B | cpu | 4 | pp512|12.08 ± 0.02| | ||
Qwen2.5 3B | 1.86 GiB | 3.40 B | cpu | 4 | tg128|2.23 ± 0.02| | ||
|
||
Q4_1 | ||
| Model | Size | Params | backend | threads | test | t/s | | ||
| -----------| -------- | ------ | ------- | ------- | ---- |------| | ||
Qwen2.5 0.5B |351.50 MiB|494.03 M| cpu | 4 | pp512|62.07 ± 0.12| | ||
Qwen2.5 0.5B |351.50 MiB|494.03 M| cpu | 4 | tg128|9.91 ± 0.01| | ||
Qwen2.5 1.5B |964.06 MiB| 1.54 B | cpu | 4 | pp512|22.95 ± 0.25| | ||
Qwen2.5 1.5B |964.06 MiB| 1.54 B | cpu | 4 | tg128|4.01 ± 0.15| | ||
Qwen2.5 3B | 1.85 GiB | 3.09 B | cpu | 4 | pp512|11.55 ± 0.16| | ||
Qwen2.5 3B | 1.85 GiB | 3.09 B | cpu | 4 | tg128|2.25 ± 0.04| | ||
|
||
|
||
Q4_K | ||
| Model | Size | Params | backend | threads | test | t/s | | ||
| -----------| -------- | ------ | ------- | ------- | ---- |------| | ||
Qwen2.5 0.5B |462.96 MiB|630.17 M| cpu | 4 | pp512|9.29 ± 0.05| | ||
Qwen2.5 0.5B |462.96 MiB|630.17 M| cpu | 4 | tg128|5.67 ± 0.04| | ||
Qwen2.5 1.5B | 1.04 GiB | 1.78 B | cpu | 4 | pp512|10.38 ± 0.10| | ||
Qwen2.5 1.5B | 1.04 GiB | 1.78 B | cpu | 4 | tg128|3.17 ± 0.08| | ||
Qwen2.5 3B | 1.95 GiB | 3.40 B | cpu | 4 | pp512|4.23 ± 0.04| | ||
Qwen2.5 3B | 1.95 GiB | 3.40 B | cpu | 4 | tg128|1.73 ± 0.00| |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.