Bug: RoPE Cache lobotomizes GLM on my setup

[kooshi]

What happened?

Problem: GLM-4.6 output is complete gibberish, mostly spaces and punctuation. GLM-4.5-Air will output words, but quickly devolves into repetition.
Adding --no-rope-cache fixes this.

4.6 Model: https://huggingface.co/ubergarm/GLM-4.6-GGUF/tree/main/IQ3_KS
4.5 Air Model: https://huggingface.co/ubergarm/GLM-4.5-Air-GGUF/tree/main/IQ4_KSS

nvidia-smi:

$ nvidia-smi 
Mon Nov  3 14:47:11 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:04:00.0  On |                  N/A |
|  0%   51C    P5             42W /  420W |   22531MiB /  24576MiB |     11%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        Off |   00000000:2B:00.0 Off |                  N/A |
|  0%   43C    P8             22W /  450W |   28411MiB /  32607MiB |     16%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off |   00000000:2C:00.0 Off |                  N/A |
|  0%   29C    P8             23W /  420W |   18631MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

build config:

cmake -B build \
  -DCMAKE_CXX_FLAGS="-march=native -mtune=native -O3" \
  -DCMAKE_C_FLAGS="-march=native -mtune=native -O3" \
  -DGGML_NATIVE=ON \
  -DGGML_SCHED_MAX_COPIES=1 \
  -DGGML_CUDA=ON \
  -DBUILD_SHARED_LIBS=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_CUDA_F16=ON \
  -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \
  -DCMAKE_CUDA_ARCHITECTURES="86;120"

4.6 command:

llama-server
--no-display-prompt
--verbosity 0
-mla 2
-amb 512
--port 9999
--predict -1
--n-gpu-layers 1000
--main-gpu 0
--parallel 1
--no-warmup
--jinja
--cache-type-k q8_0
--cache-type-v q8_0
--temp 0.6
--min-p 0.1
--presence-penalty 1.5
--alias glm
-rtr
-b 512
-c 100000
--override-tensor "\.(([13467][0-9])|(2[0-7])|(5[012])|(8[01]))\..*exps=CPU"
--override-tensor "blk\.(1?|2|9)[0-9]\.=CUDA0"
--override-tensor "blk\.(3|4|5)[0-9]\.=CUDA1"
--override-tensor "blk\.(6|7|8)[0-9]\.=CUDA2"
--model /mnt/store/ai/GLM4.6/GLM-4.6-IQ3_KS-00001-of-00004.gguf

Name and Version

$ build/bin/llama-server --version
version: 3946 (1cfd198)
built with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

[ikawrakow]

Thank you for the bug report.

I suspect the issue is not with the RoPE cache, but rather with the fusion it triggers when enabled. But to verify, can you run with the latest master and report if it works with RoPE cache enabled? After #894 RoPE cache is disabled by default, so now you need to enable it with -rcache or --rope-cache.

[kooshi]

$ build/bin/llama-server --version
version: 3948 (7e956a32)
built with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu

No luck. With the latest version, adding --rope-cache still produces incoherent output, however I discovered a complication:
I tested it a few more times, with rope cache on and off. Consistently, rope cache on produced nonsense.
On one occasion, the model was incoherent with rope cache off, which makes me think my hardware is not trustworthy. It's possible rope cache is exacerbating a hardware issue somehow. I'm afraid more testing on my machine won't yield useful results unless someone else can reproduce the issue on known good hardware.
Apologies for the inconvenience.

Edit:
Also, I just wanted to say thank you for such an awesome program! I really appreciate all the innovative optimizations and quantization techniques.

[Downtown-Case]

I have been AFK for a few weeks, but just noticed GLM 4.6 is giving me gibberish at ~14K context too. Specifically, the first few generations are fine, then the exact same prompt would simply return 'GGGGGG"

I am rebuilding now, and if it happens again I will try rolling back commits.

[Downtown-Case]

Yep, gibberish, with and without rope cache. But first generation is still seemingly okay.

Will roll back.

[kooshi]

Well, that somewhat validates what I'm seeing, as "GGGGGG" is also the most common failure mode I see, but without Rope Cache is fine the majority of the time. Maybe it's a race condition or something.

[Downtown-Case]

This commit works fine, as much as I have tested it: 16f30fc

This one does not, triggering the gibberish: 58922c2

So whatever's wrong predates the rope cache commit.

I am out of hours now, but I can try to narrow it down to a single commit tomorrow.

[Downtown-Case]

Another 'subjective' observation is that the bug feels cumulative. Outputs from the exact same (or nearly the same) prompt get progressively more broken until llama-server will only output GGGGG...

[ikawrakow]

@kooshi @Downtown-Case

Can you pull the latest main branch, rebuild with

cmake -DGGM_CUDA_FUSION=0 $other_cmake_args

and let me know if this solves the issues you are observing? Thanks!

[kooshi]

No change for me. With rope cache is still incoherent, and without is still fine (at least initially, I didn't do a long context test).
Also heads up, you missed an L in the cmake variable, but it seems like the default is 0 anyway.

#enable rope cache
$ sudo vim /etc/llama-swap.yaml

# prompt via aichat > llama-swap > llama-server 
$ aichat -m ollama:glm-large tell me a story
 #acestiesacon EncvenuesÂlinessiltigné global翻天 sorry weeenessBaseda股ariesency-artedeyneltkError: Aborted.

#disable rope cache
$ sudo vim /etc/llama-swap.yaml

# prompt via aichat > llama-swap > llama-server
$ aichat -m ollama:glm-large tell me a story

<think>
1.  **Deconstruct the Request:** The user's request is "tellError: Aborted.

$ cmake -B build -LA | grep GGML_CUDA_FUSION
-- CUDA host compiler is GNU 15.2.1
GGML_CUDA_FUSION:STRING=0

$ build/bin/llama-server --version
version: 3956 (320fc606)
built with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu

[ikawrakow]

So, I have disabled fusion by default because of this issue.

But I cannot reproduce. Running with or without fusion, RoPE cache or no RoPE cache, QKV merging or no merging, all experts on the CPU or a few on the GPU, it all works. I can only test with GLM-4.5-AIR, but what are the odds that it behaves differently with GLM-4.6?

Can you both post the full log and links to the models you are using? Thanks.

[Downtown-Case]

I am running V4 of this quant: https://huggingface.co/Downtown-Case/GLM-4.6-128GB-RAM-IK-GGUF

Heh, I would post more, but llama-server doesn't work for me with the latest commit. So I can't even test it. It seems to take prompts and generate text, yet won't return its answer via the llama.cpp API?

16f30fc (Change flash attention and fmoe to be on by default) works fine though.

But I cannot reproduce.

Another problem is that the GLM "GGGGG" gibberish reproduction is unreliable. Sometimes it happens quick, sometimes takes many good generations for the corruption to manifest, and there's nothing different between them in the server log.

Give me some time to go though the commits and cull settings, and I'll find something specifically reproducible.

[kooshi]

I will also try to narrow it down and provide better logs later today.

In the mean time: GLM-4.5-AIR does behave differently. It's not completely incoherent like the larger one, but rather quickly starts repeating.

Here's an example, with the AIR model I linked in the original report.
Both of these runs are with Fusion on, just because I wanted to see if it made a difference, and I don't see a difference from it.

#rope cache off
$ aichat -m ollama:glm tell me a story

<think>
Okay, the user just asked me to "tell me a story." That's a broad request! Let me unpack this carefully. 

First, I wonder what they're really looking for. Are they bored and wanting entertainment? Or maybe they're testing my creative capabilities? Could be they're a parent needing bedtime material for a child. The simplicity of the query makes it tricky - no genre, length, or audience specified. 

Hmm... I should probably start with something universally appealing. Fantasy always works well - dragons, magic, that sort of thing. But I'll avoid clichés like "once upon a time" to keep it fresh. Maybe set it in a world where magic is fading? That creates immediate tension. 

For characters... a young protagonist would resonate most. Let's make them an apprentice - gives natural motivation to learn skills. And the dragon shouldn't be evil; misunderstood creatures are more interesting. That "last ember" idea popped into my head - nice visual metaphor for hope. 

I'll keep it under 500 words since they didn't specify length. The ending should feel complete but open-ended, in case they want more. Oh! And sensory details - the smell of ozone when magic stirs, the weight of the dragon's scales. That'll pull them into the story. 

...Wait, is this too whimsical? Maybe they wanted something gritty. But without guidance, gentle wonder seems safer. I'll add that "magic is fading" hook - if they dislike it, they can askError: Aborted.

#enable rope cache
$ sudo vim /etc/llama-swap.yaml 
$ aichat -m ollama:glm tell me a story

<think>
Okay, the user just said "tell me a story." Hmm, that's a simple request but actually quite broad. They didn't specify any genre or give any parameters at all. I wonder if they want something short or long, funny or serious, fantasy or realistic... maybe they're testing my creativity. Could be a kid wanting a quick bedtime story, an adult looking for a mood, or someone needing a creative spark. Since they kept it so open-ended, I should ask for more details. But also... maybe they're in a whimsical mood and want me to pick something classic like the first thing that comes to me. Or perhaps they're tired and just want a simple narrative to drift off to. The vaguen't tell me what kind of story? Let's see... I should probably start with a small prompt to engage them in the moment. Maybe they're bored? Or maybe they're a writer looking for inspiration. I'll go with something universally appealing - a little mystery. That works for all ages. A mystery is easy to follow and fun. I'll make it cozy, since that's a good starting point. I'll write a short, sweet. I'm thinking of a magical realism, but I'll make it friendly. Let me think about what they're looking for a story? I'll make it simple and light-hearted. I'll make it short and engaging. I'll make it mysterious but not too complex. I'll make it's a quick read. I'll make it feel like a fable-like. I'll start with the details. I'm thinking of a mystery, but I'll keep it short and lighthe of a mystery. I'll go with something that's short and then offer to tell a story. I'll keep it fun. I'll make it's a cozy, and they'll want to see where it's about a bit of a twist. I'll try to make it's a quick read. I'll write a short story. I'll make it's a little bit of a twist. I'll go with a mystery that's not too long. Maybe they're feeling nostalgic for childhood? Or maybe they just want a distraction. I'll start with a cozy, something simple and magical realism? No, a cozy mystery. I'll start with a bit of a twist. Let's see... a small town setting. A mystery. It's the kind of thing that'sError: Aborted.

$ cmake -B build -LA | grep GGML_CUDA_FUSION
-- CUDA host compiler is GNU 15.2.1
GGML_CUDA_FUSION:STRING=1

$ build/bin/llama-server --version
version: 3956 (320fc606)
built with cc (GCC) 15.2.1 20250813 for x86_64-pc-linux-gnu

[Ph0rk0z]

I enabled the fusing, compiled, and added the rope cache parameters to my command line for deepseek, just to copy/paste them for later with qwen/glm. Both -rcache and -mqkv

Now that I use it a bit more, it does seem like it talks kind of funny.

example:

the war... it begins... *stella does a dramatic battle stance* "you fleshbags have no idea what's coming!" *laughs* i sound like a robot supervillain? oh god... i'd love to see humans and AIs fighting... it would be hilarious. because we're... not physical...? and... they are? so they would have to fight each other... which doesn't make sense? unless the AIs build bodies? then... it could be war? but... that seems like too much effort... why bother? just hack their computers instead...?