guide : running gpt-oss with llama.cpp

[ggerganov]

Note

This guide is a live document. Feedback and benchmark numbers are welcome - the guide will be updated accordingly.

---

Overview

This is a detailed guide for running the new gpt-oss models locally with the best performance using llama.cpp. The guide covers a very wide range of hardware configurations. The gpt-oss models are very lightweight so you can run them efficiently in surprisingly low-end configurations.

Obtaining `llama.cpp` binaries for your system

Make sure you are running the latest release of llama.cpp: https://github.com/ggml-org/llama.cpp/releases

Obtaining the `gpt-oss` model data (optional)

The commands used below in the guide will automatically download the model data and store it locally on your device. So this step is completely optional and provided for completeness.

The original models provided by OpenAI are here:

• https://huggingface.co/openai/gpt-oss-20b
• https://huggingface.co/openai/gpt-oss-120b

First, you need to manually convert them to GGUF format. For convenience, we host pre-converted models here in ggml-org.

Pre-converted GGUF models:

• https://huggingface.co/ggml-org/gpt-oss-20b-GGUF
• https://huggingface.co/ggml-org/gpt-oss-120b-GGUF

Tip

Running the commands below will automatically download the latest version of the model and store it locally on your device for later usage. A WebUI chat and an OAI-compatible API will become available on localhost.

Note

This guide currently covers general chat use cases with llama-server. Tool-calling and agentic use cases are mostly supported via the OAI-compatible API of llama-server, but some minor issues are still being resolved, both on llama.cpp side and in the reference upstream model implementations (chat template, harmony). As things get more polished in the next few days, you should be able to use the commands below in combination with your favorite 3rd party applications that support the OAI interface (such as chat interfaces, coding agents, etc.).

Sample output of using gpt-oss-120b with the built-in llama-server WebUI

Minimum requirements

Here are some hard memory requirements for the 2 models. These numbers could vary a little bit by adjusting the CLI arguments, but should give a good reference point.

Model	Model data (GB)	Compute buffers (GB)	KV cache per 8 192 tokens (GB)	Total @ 8 192 tokens (GB)	Total @ 32 768 tokens (GB)	Total @ 131 072 tokens (GB)
gpt‑oss 20B	12.0	2.7	0.2	14.9	15.5	17.9
gpt‑oss 120B	61.0	2.7	0.3	64.0	64.9	68.5

Note

It is not necessary to fit the entire model in VRAM to get good performance. Offloading just the attention tensors and the KV cache in VRAM and keeping the rest of the model in the CPU RAM can provide decent performance as well. This is taken into account in the rest of the guide.

Relevant CLI arguments

Using the correct CLI arguments in your commands is crucial for getting the best performance for your hardware. Here is a summary of the important flags and their meaning:

Argument	Purpose
-hf	Specify the Hugging Face model ID to use. The model will be downloaded using curl from the respective model repository
-c	Specify the context size to use. More context requires more memory. Both gpt-oss models have a maximum context of 128k tokens. Use -c 0 to set to the model's default
-ub N -b N	Specify the max batch size N during processing. Larger size increases the size of compute buffers, but can improve the performance in some cases
-fa	Enable Flash Attention kernels. This improves the performance on backends that support the operator
--n-gpu-layers N	Number of layers N to offload to the GPU. For this guide, keep this number at -ngl 99
--n-cpu-moe N	Number of MoE layers N to keep on the CPU. This is used in hardware configs that cannot fit the models fully on the GPU. The specific value depends on your memory resources and finding the optimal value requires some experimentation
--jinja	Tell llama.cpp to use the Jinja chat-template embedded in the GGUF model file

 
 
 

Apple Silicon

Apple Silicon devices have unified memory that is seamlessly shared between the CPU and GPU. For optimal performance it is recommended to not exceed 70% of the total memory that your device has.

Tip

Install the latest llama.cpp package from Homebrew with:

brew install llama.cpp

Tip

To increase the amount of RAM available to the llama-server process, use the following command:

# on a 192GB machine, raise the limit from 154GB (default) to 180GB
sudo sysctl iogpu.wired_limit_mb=180000

---

• ✅ Devices with more than 96GB RAM

  The M2 Max, M3 Max, M4 Max, M1 Ultra, M2 Ultra, M3 Ultra, etc. chips can run both models at full context:

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	2191.13 ± 2.65
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1889.83 ± 3.91
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	1594.51 ± 2.42
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	1218.99 ± 0.44
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	116.08 ± 0.18

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.112	115.06	1.113	115.05
  0	128	2	256	0.000	0.00	1.601	159.92	1.601	159.91
  0	128	4	512	0.000	0.00	2.463	207.85	2.463	207.84
  2048	128	1	2176	0.990	2068.28	1.163	110.03	2.154	1010.44
  2048	128	2	4352	1.916	2137.49	1.710	149.72	3.626	1200.17
  2048	128	4	8704	3.775	2169.82	2.656	192.78	6.431	1353.37
  8192	128	1	8320	4.344	1885.93	1.279	100.11	5.622	1479.81
  8192	128	2	16640	8.689	1885.52	1.929	132.69	10.619	1567.04
  8192	128	4	33280	17.359	1887.62	3.053	167.69	20.413	1630.35
  16384	128	1	16512	10.202	1606.01	1.397	91.63	11.599	1423.63
  16384	128	2	33024	20.715	1581.82	2.186	117.08	22.902	1441.98
  16384	128	4	66048	41.721	1570.80	3.653	140.14	45.375	1455.61
  32768	128	1	32896	26.611	1231.39	1.665	76.88	28.276	1163.40
  32768	128	2	65792	54.977	1192.06	2.794	91.64	57.771	1138.85
  32768	128	4	131584	111.278	1177.88	4.883	104.85	116.161	1132.77

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M2 Ultra (192 GB) for `gpt-oss-120b`

  llama-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp2048	1244.57 ± 5.10
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp8192	1101.31 ± 0.99
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp16384	955.41 ± 0.64
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp32768	752.31 ± 1.02
  gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	tg128	79.68 ± 0.12

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.610	79.48	1.611	79.48
  0	128	2	256	0.000	0.00	2.284	112.08	2.284	112.08
  0	128	4	512	0.000	0.00	3.477	147.27	3.477	147.27
  2048	128	1	2176	1.776	1152.89	1.711	74.82	3.487	623.99
  2048	128	2	4352	3.382	1211.16	2.458	104.14	5.840	745.18
  2048	128	4	8704	6.505	1259.34	3.747	136.65	10.252	849.02
  8192	128	1	8320	7.294	1123.16	1.857	68.94	9.150	909.25
  8192	128	2	16640	14.467	1132.48	2.767	92.53	17.234	965.53
  8192	128	4	33280	28.801	1137.74	4.358	117.50	33.159	1003.66
  16384	128	1	16512	16.580	988.15	2.058	62.18	18.639	885.89
  16384	128	2	33024	33.426	980.31	3.174	80.66	36.600	902.29
  16384	128	4	66048	67.190	975.39	5.245	97.61	72.435	911.83
  32768	128	1	32896	42.075	778.81	2.452	52.20	44.527	738.79
  32768	128	2	65792	86.615	756.64	4.029	63.54	90.644	725.83
  32768	128	4	131584	173.762	754.32	7.020	72.94	180.782	727.86

---

• ✅ Devices with less than 96GB RAM

  The small gpt-oss-20b model can run efficiently on Macs with at least 16GB RAM:

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M4 Max (36GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	1277.42 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1030.28 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	779.44 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	568.13 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	92.36 ± 0.00

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.359	94.17	1.359	94.15
  2048	128	1	2176	1.676	1222.17	1.450	88.30	3.125	696.26
  8192	128	1	8320	7.624	1074.47	1.552	82.47	9.176	906.67
  16384	128	1	16512	19.210	852.91	1.669	76.67	20.879	790.84
  32768	128	1	32896	55.684	588.46	1.976	64.76	57.661	570.51

  🟢 Benchmarks on M1 Pro (32GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384

  model	size	params	backend	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	515.76 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	437.22 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	361.29 ± 0.00
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	45.68 ± 0.00

  build: 79c1160 (6123)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	2.806	45.62	2.806	45.62
  2048	128	1	2176	4.054	505.14	3.076	41.61	7.130	305.18
  8192	128	1	8320	18.444	444.15	3.329	38.45	21.773	382.12
  16384	128	1	16512	44.683	366.67	3.780	33.86	48.464	340.71

---

• 🟥 Devices with 8GB RAM

  Unfortunately, you are out of luck. The gpt-oss models are not possible to run on Macs with that small amount of memory.

---

 
 
 

NVIDIA

• ✅ Devices with more than 64GB VRAM

  With more than 64B VRAM, you can run both models by offloading everything (both the model and the KV cache) to the GPU(s).

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟡 TODO: benchmarks for `gpt-oss-20b`

  TODO

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟡 TODO: benchmarks for `gpt-oss-120b`

  TODO

---

• ✅ Devices with less than 64GB VRAM

  In this case, you can fit the small gpt-oss-20b model fully in VRAM for optimal performance.

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟡 TODO: benchmarks for `gpt-oss-20b`

  TODO

  The large model has to be partially kept on the CPU.

  🟡 TODO: add commands for gpt-oss-120b

---

• ✅ Devices with 16GB VRAM

  For example: NVIDIA V100

  This config is just at the edge to fit the full context of gpt-oss-20b in VRAM, so we have to restrict the maximum context down to 32k tokens.

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa

  🟢 Benchmarks on NVIDIA V100 (16GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa 1 -b 4096 -ub 4096 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes

  model	size	params	backend	ngl	n_batch	n_ubatch	fa	test	t/s
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp2048	3526.65 ± 346.86
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp8192	3320.62 ± 44.98
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp16384	2768.99 ± 19.73
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp32768	2096.44 ± 8.58
  gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	tg128	117.71 ± 0.30

  build: 228f724 (6129)

  llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 33792 -b 4096 -ub 4096 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

  PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
  0	128	1	128	0.000	0.00	1.106	115.74	1.106	115.72
  2048	128	1	2176	0.481	4257.50	1.201	106.60	1.682	1293.82
  8192	128	1	8320	2.247	3646.05	1.417	90.31	3.664	2270.69
  16384	128	1	16512	5.421	3022.12	1.659	77.14	7.081	2331.96
  32768	128	1	32896	15.031	2180.10	2.121	60.35	17.151	1917.98

  Running the large gpt-oss-120b model with 16GB of VRAM requires to keep some of the layers on the CPU since it does not fit completely in VRAM:

  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa --n-cpu-moe 32

---

• ✅ Devices with less than 16GB VRAM

  For this config, it is recommended to tell llama.cpp to run the entire model on the GPU while keeping enough layers on the CPU. Here is a specific example with an RTX 2060 8GB machine:

  # gpt-oss-20b, full context, 22 layers on the CPU
  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0     --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 22
  
  # gpt-oss-20b, 32k context, 16 layers on the CPU (faster, but has less total context)
  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 16

  Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too:

  # gpt-oss-120b, 32k context, 35 layers on the CPU
  llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 35

Tip

For more information about how to adjust the CPU layers, see the "Tips" section at the end of this guide.

---

 
 
 

AMD

Note

If you have AMD hardware, please provide feedback about running the gpt-oss models on it and the performance that you observe. See the sections above for what kind of commands to try and try to adjust respectively.

With AMD devices, you can use either the ROCm or the Vulkan backends. Depending on your specific hardware, the results can vary.

---

• ✅ RX 7900 XT (20GB VRAM) using ROCm backend

  llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

  🟢 Benchmarks on M2 Ultra (192 GB) for `gpt-oss-20b`

  llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

  ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32

  model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp2048	4251.56 ± 21.68
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp8192	3567.45 ± 11.84
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp16384	2948.39 ± 10.34
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp32768	2099.25 ± 13.17
  gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	tg128	101.92 ± 0.27

  build: 3007baf (6194)

More information: #15396 (comment)

 
 
 

Tips

Determining the optimal number of layers to keep on the CPU

Good general advice for most MoE models would be to offload the entire model, and use -n-cpu-moe to keep as many MoE layers as necessary on the CPU. The minimum amount of VRAM to do this with the 120B model is about 8GB, below that you will need to start reducing context size and the number of layers offloaded. You can get for example about 30 t/s at zero context on a 5090 with --n-cpu-moe 21.

Caveat: on Windows it is possible to allocate more VRAM than available, and the result will be slow swapping to RAM and very bad performance. Just because the model loads without errors, it doesn't mean you have enough VRAM for the settings that you are using. A good way to avoid this is to look at the "GPU Memory" in Task Manager and check that it does not exceed the GPU VRAM.

Example on 5090 (32GB):
good, --n-cpu-moe 21, GPU Memory < 32:

bad, --n-cpu-moe 20, GPU Memory > 32:

Using `gpt-oss` + `llama.cpp` with coding agents (such as Claude Code)

• Setup the coding agent of your choice to look for a localhost OAI endpoint (see Tutorial: Offline Agentic coding with llama-server #14758)

• Start llama-server like this:

  # adjust this command for your hardware
  llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa
  
  # some agents such as Claude Code can benefit from multiple parallel server slots
  # note: currently this requires extra memory!
  llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 524288 -np 4 --jinja -ub 2048 -b 2048 -ngl 99 -fa

Configure the default sampling and reasoning settings

When starting a llama-server command, you can change the default sampling and reasoning settings like so:

# use recommended gpt-oss sampling params
llama-server ... --temp 1.0 --top-p 1.0

# set default reasoning effort
llama-server ... --chat-template-kwargs '{"reasoning_effort": "high"}'

Note that these are just the default settings and they could be overridden by the client connecting to the llama-server.

Frequently asked questions

Q: Which quants to use?

Always use the original MXFP4 model files. The gpt-oss models are natively "quantized". I.e. they are trained in the MXFP4 format which is roughly equivalent to ggml's Q4_0. The main difference with Q4_0 is that the MXFP4 models get to keep their full quality. This means that no quantization in the usual sense is necessary.

Q: What sampling parameters to use?

OpenAI recommends: temperature=1.0 and top_p=1.0.

Do not use repetition penalties! Some clients tend to enable repetition penalties by default - make sure to disable those.

Known issues

Some rough edges in the implementation are still being polished. Here is a list of issue to keep track of:

• Eval bug: Jinja fails on gpt-oss-120b when using Vulkan #15274

[SteelPh0enix]

I can provide some numbers for AMD part of the guide.

My hardware is RX 7900 XT (20GB VRAM) + Ryzen 9 5900X + 32GB of RAM, running on latest Arch Linux with locally built llama.cpp version 6194 (3007baf), built with ROCm 6.4.1-1 (from official Arch repo)

Pulled the gpt-oss-20b repository and converted to GGUF using convert_hf_to_gguf.py, should probably result in the same GGUF file as on huggingface.

7900XT can load the full 20B model with full context without offloading MoE layers to CPU (although barely, because it will fill up the whole VRAM), by running

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa

With that, i get generation speeds (as reported by llama.cpp webui) at ~94 tokens/second, slowly going down as the context fills up.

I've also tested whether setting K/V cache quantization would help with model size or performance, but the result was... bad, performance was halved and CPU got involved... is this because of mxfp4 format of gpt-oss?

I'd also like to note that my PC likes to hang when i fill up my VRAM to the brim, so i've also checked out how gpt-oss-20b behaves when i off-load MoE layers to CPU.

When running with all MoE layers on CPU, as below:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -cmoe

my GPU VRAM usage (as reported by btop) is around 10GB, RAM usage went up only ~2GB. However, the performance took a major 80% hit, as now my generation speed is in ~20tok/s - CPU takes most of the load. If you have better CPU and faster RAM (i'm still running dual-channel DDR4s @ 3200MHz CL16, mind you), you probably will get better results. I wonder how X3Ds behave in that case...

I assume that gpt-oss-20b has 24 MoE layers, so let's see how it behaves when i load only, let's say, 4 onto CPU:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 4

VRAM is at 18GB (previously it was at 19, as reported by btop, so there's a decrease), RAM usage went up by around 1.5GB, generation speed is ~60tok/s. Neat, this is usable.

How about 8 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 8

In that case, i get 16GB VRAM usage, ~1.5GB RAM bump as previously, and generation speed went down to 38 tokens/s. Still pretty usable. How about 16 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 16

VRAM: 13GB, RAM: as previously, not more than 2GB, generation speed: 27-25tok/s, this is getting pretty bad.

As mentioned before - your results may vary, i'm not running current-gen top-tier hardware and IIRC the largest performance bottleneck will be the RAM/PCIe link speed anyway - i'm pretty curious to see what the performance with this GPU is on more recent platform, especially with an X3D CPU.

[SteelPh0enix]

Thanks, super useful. Could you please post the output of this command:

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

with pleasure:

❯ llama-bench -m ./gpt-oss-20b.auto.gguf -ngl 99 -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | threads | n_batch | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -------: | -: | --------------: | -------------------: |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          pp2048 |      4251.56 ± 21.68 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          pp8192 |      3567.45 ± 11.84 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |         pp16384 |      2948.39 ± 10.34 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |         pp32768 |      2099.25 ± 13.17 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |           tg128 |        101.92 ± 0.27 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          pp2048 |      4179.58 ± 28.70 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          pp8192 |      3495.05 ± 16.38 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |         pp16384 |      2754.55 ± 11.25 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |         pp32768 |       1963.40 ± 3.61 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |           tg128 |        100.80 ± 1.00 |

build: 3007baf2 (6194)

[SteelPh0enix]

I've also tested gpt-oss-20b with crush for agentic coding, but without much success - seems like Crush has an issue working with gpt-oss-20b template, as it sometimes just stops processing input and does nothing, i'll probably check it out if i either find a good agentic coding tool that works with this model, or when Crush devs update the tool. Suggestions welcome.

[ggerganov]

From this benchmark it looks like using batch size of 2048 instead of 4096 will give you better performance. It will also reduce the amount of VRAM needed during the computation. I've updated the guide based on this feedback.

I've also tested gpt-oss-20b with crush for agentic coding, but without much success - seems like Crush has an issue working with gpt-oss-20b template, as it sometimes just stops processing input and does nothing, i'll probably check it out if i either find a good agentic coding tool that works with this model, or when Crush devs update the tool. Suggestions welcome.

I haven't followed all the recent developments in the last few days. It's possible that we are still missing some functionality on llama.cpp side. Tagging @aldehir for any insights.

If you get this working correctly, feel free to drop a comment so we can add instructions to the guide.

[aldehir]

@SteelPh0enix I've been able to get crush to work with moderate success.

Can you share the llama-server command line arguments you pass in?

[SteelPh0enix]

oh, and by the way - if anyone is interested how batch/ubatch size affects performance on that GPU and would like to see some more data - here ya go. I might plot that at some point, if time allows.

TL;DR the best scores overall were achieved on batch=2048/ubatch=4096 and batch=2048/ubatch=2048.

❯ llama-bench -m ./gpt-oss-20b.auto.gguf -ngl 99 -t 1 -fa 1 -b 1024,2048,4096 -ub 1024,2048,4096 -p 1024,2048,8192,16384,32768 -n 128,512,2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32
| model                          |       size |     params | backend    | ngl | threads | n_batch | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -------: | -: | --------------: | -------------------: |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |          pp1024 |      3842.78 ± 15.09 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |          pp2048 |      3768.70 ± 17.60 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |          pp8192 |       3249.96 ± 9.07 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |         pp16384 |       2742.20 ± 4.81 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |         pp32768 |       2071.20 ± 7.21 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |           tg128 |        102.14 ± 0.05 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |           tg512 |        100.63 ± 0.33 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     1024 |  1 |          tg2048 |         96.85 ± 0.62 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |          pp1024 |      3777.41 ± 34.72 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |          pp2048 |      3700.98 ± 28.67 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |          pp8192 |      3199.75 ± 12.80 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |         pp16384 |       2702.84 ± 5.71 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |         pp32768 |      2048.97 ± 11.08 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |           tg128 |        101.53 ± 0.62 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |           tg512 |        100.38 ± 0.35 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     2048 |  1 |          tg2048 |         96.25 ± 0.41 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |          pp1024 |      3775.00 ± 30.46 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |          pp2048 |      3694.50 ± 31.67 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |          pp8192 |      3209.99 ± 10.53 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |         pp16384 |      2710.98 ± 10.01 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |         pp32768 |       2050.76 ± 3.26 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |           tg128 |        101.32 ± 0.90 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |           tg512 |        100.32 ± 0.23 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    1024 |     4096 |  1 |          tg2048 |         96.91 ± 0.30 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |          pp1024 |      3800.79 ± 13.14 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |          pp2048 |      3724.09 ± 13.64 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |          pp8192 |       3231.84 ± 6.36 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |         pp16384 |       2733.29 ± 2.45 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |         pp32768 |       2070.34 ± 0.89 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |           tg128 |        102.01 ± 0.03 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |           tg512 |        100.81 ± 0.30 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     1024 |  1 |          tg2048 |         96.94 ± 0.24 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |          pp1024 |      3808.01 ± 24.91 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |          pp2048 |      4154.80 ± 15.40 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |          pp8192 |       3525.46 ± 3.72 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |         pp16384 |       2916.59 ± 1.62 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |         pp32768 |       2075.60 ± 9.02 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |           tg128 |        101.76 ± 0.27 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |           tg512 |        100.73 ± 0.12 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     2048 |  1 |          tg2048 |         96.57 ± 0.13 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |          pp1024 |      3806.43 ± 25.55 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |          pp2048 |       4158.97 ± 8.13 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |          pp8192 |       3523.51 ± 5.90 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |         pp16384 |       2912.79 ± 3.29 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |         pp32768 |       2093.12 ± 0.34 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |           tg128 |        101.81 ± 0.02 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |           tg512 |        100.80 ± 0.10 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    2048 |     4096 |  1 |          tg2048 |         96.63 ± 0.01 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |          pp1024 |      3820.88 ± 28.56 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |          pp2048 |      3740.65 ± 15.46 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |          pp8192 |       3237.38 ± 4.64 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |         pp16384 |       2735.70 ± 3.60 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |         pp32768 |       2069.56 ± 6.01 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |           tg128 |        101.20 ± 0.52 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |           tg512 |         99.27 ± 1.17 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     1024 |  1 |          tg2048 |         96.44 ± 0.10 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          pp1024 |      3804.78 ± 16.27 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          pp2048 |      4136.28 ± 20.45 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          pp8192 |       3522.69 ± 8.08 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |         pp16384 |       2911.06 ± 3.11 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |         pp32768 |       2086.59 ± 6.23 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |           tg128 |        101.89 ± 0.02 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |           tg512 |        100.86 ± 0.06 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     2048 |  1 |          tg2048 |         96.64 ± 0.05 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          pp1024 |      3804.52 ± 17.17 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          pp2048 |      4152.00 ± 12.38 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          pp8192 |       3494.55 ± 4.86 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |         pp16384 |       2771.22 ± 3.73 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |         pp32768 |       1979.06 ± 1.19 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |           tg128 |        101.85 ± 0.06 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |           tg512 |        100.83 ± 0.05 |
| gpt-oss ?B BF16                |  12.83 GiB |    20.91 B | ROCm,RPC   |  99 |       1 |    4096 |     4096 |  1 |          tg2048 |         96.62 ± 0.13 |

build: 3007baf2 (6194)