Skip to content

Commit 5e90233

Browse files
authored
opencl: update doc (ggml-org#17011)
* opencl: update docs * opencl: update docs * opencl: fix link * opencl: update doc
1 parent a5c07dc commit 5e90233

File tree

1 file changed

+25
-3
lines changed

1 file changed

+25
-3
lines changed

docs/backend/OPENCL.md

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,25 +39,41 @@ The llama.cpp OpenCL backend is designed to enable llama.cpp on **Qualcomm Adren
3939
| Adreno 830 (Snapdragon 8 Elite) | Support |
4040
| Adreno X85 (Snapdragon X Elite) | Support |
4141

42+
> A6x GPUs with a recent driver and compiler are supported; they are usually found in IoT platforms.
43+
However, A6x GPUs in phones are likely not supported due to the outdated driver and compiler.
44+
4245
## DataType Supports
4346

4447
| DataType | Status |
4548
|:----------------------:|:--------------------------:|
4649
| Q4_0 | Support |
4750
| Q6_K | Support, but not optimized |
51+
| Q8_0 | Support |
52+
| MXFP4 | Support |
4853

4954
## Model Preparation
5055

51-
You can refer to the general [*Prepare and Quantize*](README.md#prepare-and-quantize) guide for model prepration.
56+
You can refer to the general [llama-quantize tool](/tools/quantize/README.md) for steps to convert a model in Hugging Face safetensor format to GGUF with quantization.
5257

53-
Currently we support `Q4_0` quantization and have optimize for it. To achieve best performance on Adreno GPU, add `--pure` to `llama-quantize`. For example,
58+
Currently we support `Q4_0` quantization and have optimized for it. To achieve best performance on Adreno GPU, add `--pure` to `llama-quantize` (i.e., make all weights in `Q4_0`). For example,
5459

5560
```sh
5661
./llama-quantize --pure ggml-model-qwen2.5-3b-f16.gguf ggml-model-qwen-3b-Q4_0.gguf Q4_0
5762
```
5863

5964
Since `Q6_K` is also supported, `Q4_0` quantization without `--pure` will also work. However, the performance will be worse compared to pure `Q4_0` quantization.
6065

66+
### `MXFP4` MoE Models
67+
68+
OpenAI gpt-oss models are MoE models in `MXFP4`. The quantized model will be in `MXFP4_MOE`, a mixture of `MXFP4` and `Q8_0`.
69+
For this quantization, there is no need to specify `--pure`.
70+
For gpt-oss-20b model, you can directly [download](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) the quantized GGUF file in `MXFP4_MOE` from Hugging Face.
71+
72+
Although it is possible to quantize gpt-oss-20b model in pure `Q4_0` (all weights in `Q4_0`), it is not recommended since `MXFP4` has been optimized for MoE while `Q4_0` is not. In addition, accuracy should degrade with such pure `Q4_0` quantization.
73+
Hence, using the default `MXFP4_MOE` quantization (see the link above) is recommended for this model.
74+
75+
> Note that the `Q4_0` model found [here](https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-Q4_0.gguf) is a mixture of `Q4_0`, `Q8_0` and `MXFP4` and gives better performance than `MXFP4_MOE` quantization.
76+
6177
## CMake Options
6278

6379
The OpenCL backend has the following CMake options that control the behavior of the backend.
@@ -146,10 +162,13 @@ A Snapdragon X Elite device with Windows 11 Arm64 is used. Make sure the followi
146162
* Ninja
147163
* Visual Studio 2022
148164
* Powershell 7
165+
* Python
149166

150167
Visual Studio provides necessary headers and libraries although it is not directly used for building.
151168
Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.
152169

170+
> Note that building using Visual Studio's cl compiler is not supported. Clang must be used. Clang depends on libraries provided by Visual Studio to work. Therefore, Visual Studio must be installed. Alternatively, Visual Studio Build Tools can be installed instead of the full Visual Studio.
171+
153172
Powershell 7 is used for the following commands.
154173
If an older version of Powershell is used, these commands may not work as they are.
155174

@@ -201,9 +220,12 @@ ninja
201220

202221
## Known Issues
203222

204-
- Currently OpenCL backend does not work on Adreno 6xx GPUs.
223+
- Flash attention does not always improve performance.
224+
- Currently OpenCL backend works on A6xx GPUs with recent drivers and compilers (usually found in IoT platforms).
225+
However, it does not work on A6xx GPUs found in phones with old drivers and compilers.
205226

206227
## TODO
207228

208229
- Optimization for Q6_K
209230
- Support and optimization for Q4_K
231+
- Improve flash attention

0 commit comments

Comments
 (0)