Eval bug: SIGSEGV GPT-OSS:20B

[mhitza]

Name and Version

$ ./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX 4000 SFF Ada Generation, compute capability 8.9, VMM: yes
version: 6145 (04e1626)
built with gcc-13 (SUSE Linux) 13.3.1 20250313 [revision 4ef1d8c84faeebffeb0cc01ee22e891b41e5c4e0] for x86_64-suse-linux

Operating systems

Linux

GGML backends

CUDA

Hardware

13th Gen Intel(R) Core(TM) i5-13500 & NVIDIA RTX 4000 SFF Ada Generation

Models

gpt-20B.mxfp4 ( https://huggingface.co/ggml-org/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-mxfp4.gguf )

Problem description & steps to reproduce

Server is started with the following arguments

$ llama-server -m gpt-20B.mxfp4.gguf \
    --no-webui --numa numactl --threads 32 --ctx-size 131072 \
    --n-gpu-layers 37 --no-op-offload \
    --temp 0 \
    --flash-attn \
    --jinja --chat-template-kwargs '{"reasoning_effort": "high"}' --reasoning-format none  \
    --repeat-penalty 1.2

Pass in the following prompt

curl -s -N -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{ \"messages\": [{\"role\": \"user\", \"content\": \"# apache forward a request to two backend in order an cgi program then a php fcgi call\"}], \"stream\":true }"

Then after about ~2 minutes on this system (at around 70 tokens/second) it crashes. Memory stays steady during that time (watching over in nvidia-smi) at around 14GB (of 20GB available).

First Bad Commit

8b2aac0 is the first bad commit

(previous commit sha reported is what version I was running when starting to draft the report)

Relevant log output

systemd-coredump[22807]: [🡕] Process 22624 (llama-server) of user 1000 dumped core.
                                                                  
Stack trace of thread 22624:
#0  0x00000000005bd7ec n/a (/home/developer/llama.cpp/source/build/bin/llama-server + 0x1bd7ec)
ELF object binary architecture: AMD x86-64


### in gdb

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005bd7ec in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, st
--Type <RET> for more, q to quit, c to continue without paging--c
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.38-150600.14.32.1.x86_64 krb5-debuginfo-1.20.1-150600.11.11.2.x86_64 libcom_err2-debuginfo-1.47.0-150600.4.6.2.x86_64 libcurl4-debuginfo-8.6.0-150600.4.21.1.x86_64 libgcc_s1-debuginfo-14.2.0+git10526-150000.1.6.1.x86_64 libgomp1-debuginfo-14.2.0+git10526-150000.1.6.1.x86_64 libopenssl3-debuginfo-3.1.4-150600.5.33.1.x86_64 libsasl2-3-debuginfo-2.1.28-150600.7.6.2.x86_64 libssh4-debuginfo-0.9.8-150600.11.3.1.x86_64 libstdc++6-debuginfo-14.2.0+git10526-150000.1.6.1.x86_64 nvidia-compute-G06-debuginfo-580.65.06-1.x86_64d::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()
[Current thread is 1 (Thread 0x7f395d63d000 (LWP 22624))]

[mhitza]

Sorry for hijacking this project to report an issue found for an open PR, but I found it easier and more transparent in this repository #15181

@aldehir thanks for your work on the harmony parser, I'm testing out the model on top of your open branch where I've found the SIGSEGV

[aldehir]

Does it happen pretty often?

[mhitza]

It sigsegv's every time with the example prompt (since I'm running at temp 0). Cannot make a general estimate since I'm running less than a dozen prompts a day.

[mallorbc]

I have periodically been compiling llama.cpp from #15181 to deloy and use GPT-OSS-120B with Claude Code and Qwen Code.

Periodically, I have also seen this issue. Not sure what is causing it, but just wanted to share I am also regularly seeing this issue as well.

I have not ran llama.cpp from the latest commit of the PR though

[mhitza]

Based on the first bad commit and the dump that point at regexes I'm going to guess that the issue is with the regex that tries to capture the <|end|> token.

Given that in this scenario the other symptom is that reasoning runs for a long time. (As mentioned before, takes around 2 minutes, while most other LLM prompts it takes on average around 1 minute to "think"). If that's the case removing the repetition penalty would make it possible to trigger the error even more often (as in my experience it's easy to have the model loop)

d::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs(std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode, long) ()

[ggerganov]

Using the latest version of #15181, I can't reproduce using commands in OP, neither on Metal, nor with the RTX 2060 (with --cpu-moe).

@mhitza Since you have a repro can you build in debug and provide a backtrace when the crash happens? Just in case update to the latest branch.

[mhitza]

@ggerganov here's the stacktrace gist.

In not a C++ developer so...

I'm missing some distro debuginfo packages (for cuda among others) which I'm still trying to wrap my head around to provide a better backtrace, but this is a partial context when I open it up in gdb the binary with the core dump.

Core was generated by `/home/developer/llama.cpp/source/build/bin/llama-server -m /home/developer/mode'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005c2314 in std::_Function_handler<bool (char), std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false> >::_M_invoke(std::_Any_data const&, char&&) (

    
--Type <RET> for more, q to quit, c to continue without paging--
__functor=@0x17d211e0: {_M_unused = {_M_object = 0x11298570, _M_const_object = 0x11298570, _M_function_pointer = 0x11298570, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x11298570}, _M_pod_data = "p\205)\021\000\000\000\000\000\000\000\000\000\000\000"}, __args#0=@0x7ffc57eef034: 108 'l')
  at /usr/include/c++/13/bits/std_function.h:290
290             return std::__invoke_r<_Res>(*_Base::_M_get_pointer(__functor),
[Current thread is 1 (Thread 0x7fcd7fd41000 (LWP 14717))]
(gdb) list
285           }
286
287           static _Res
288           _M_invoke(const _Any_data& __functor, _ArgTypes&&... __args)
289           {
290             return std::__invoke_r<_Res>(*_Base::_M_get_pointer(__functor),
291                                          std::forward<_ArgTypes>(__args)...);
292           }
293
294           template<typename _Fn>
(gdb) 

Would providing the coredump be helpful? Or do you need to also have the binary built in my exact system (exactly the same library and compiler versions?)

[slaren]

@mhitza it would be useful if you run bt in gdb to obtain a backtrace. Ideally with a debug build, using -DCMAKE_BUILD_TYPE=Debug.

[mhitza]

@slaren thanks for the info

(gdb) bt
#0  0x00000000005c2314 in std::_Function_handler<bool (char), std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false> >::_M_invoke(std::_Any_data const&, char&&) (
    __functor=@0x17d211e0: {_M_unused = {_M_object = 0x11298570, _M_const_object = 0x11298570, _M_function_pointer = 0x11298570, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x11298570}, _M_pod_data = "p\205)\021\000\000\000\000\000\000\000\000\000\000\000"}, __args#0=@0x7ffc57eef034: 108 'l') at /usr/include/c++/13/bits/std_function.h:290
#1  0x00000000005c4f42 in std::function<bool (char)>::operator()(char) const (this=0x17d211e0, __args#0=108 'l') at /usr/include/c++/13/bits/std_function.h:591
#2  0x00000000005bc60c in std::__detail::_State<char>::_M_matches (this=0x17d211d0, __char=108 'l') at /usr/include/c++/13/bits/regex_automaton.h:170
#3  0x0000000000727b21 in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_handle_match (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=20) at /usr/include/c++/13/bits/regex_executor.tcc:327
#4  0x00000000007262ea in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=20) at /usr/include/c++/13/bits/regex_executor.tcc:530
#5  0x0000000000729571 in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_rep_once_more (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=21) at /usr/include/c++/13/bits/regex_executor.tcc:185
#6  0x000000000072773d in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_handle_repeat (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=21) at /usr/include/c++/13/bits/regex_executor.tcc:213
#7  0x0000000000726230 in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=21) at /usr/include/c++/13/bits/regex_executor.tcc:516
#8  0x0000000000727b4f in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_handle_match (this=0x7ffc586e6760, 
    __match_mode=std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_Match_mode::_Exact, __i=20) at /usr/include/c++/13/bits/regex_executor.tcc:330
#9  0x00000000007262ea in std::__detail::_Executor<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<std::reverse_iterator<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, std::__cxx11::regex_traits<char>, true>::_M_dfs (this=0x7ffc586e6760, 

/usr/include/c++/13/bits/regex_automaton.h

162       // Since correct ctor and dtor rely on _M_opcode, it's better not to
163       // change it over time.
164       _Opcode
165       _M_opcode() const noexcept
166       { return _State_base::_M_opcode; }
167 
168       bool
169       _M_matches(_Char_type __char) const
170       { return _M_get_matcher()(__char); }
171 
172       _MatcherT&
173       _M_get_matcher() noexcept
174       { return *static_cast<_MatcherT*>(this->_M_matcher_storage._M_addr()); }
175 
176       const _MatcherT&
177       _M_get_matcher() const noexcept
178       {
179         return *static_cast<const _MatcherT*>(
180             this->_M_matcher_storage._M_addr());
181       }
182     };

[aldehir]

Can you run with -v and report back the output the model generates right before it crashes.

[slaren]

@mhitza that doesn't look like the full output of bt, it seems that it only printed the first 10 entries. We need the full output, that includes the llama.cpp code that called these functions.

[aldehir]

@mhitza try running bt 100 to get more lines in your output.

[mhitza]

@aldehir @slaren first 1000 lines of the bt.

I ran the following command to get the dump gdb -ex "bt full" -ex "quit" source/build/bin/llama-server coredumpfile > backtrace.log. I stopped the process when the backtrace got already 2GB large.

Thousands of lines of std::reverse_iterator calls but with different parameters. Then at around like 850k the triggering regex call and input.

A couple of lines of snippet of the system log before the crash.

aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]: LoadModule cgid_module modules/mod_cgid.so
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]: <VirtualHost *:80>
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:     ServerName example.com
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:     # First, handle CGI scripts (e.g., /cgi-bin/*)
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:     ScriptAlias /cgi-bin/ "/usr/lib/cgi-bin/"
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:     <Directory "/usr/lib/cgi-bin">
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:         AllowOverride None
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:         Options +ExecCGI -MultiViews
aug 13 21:34:18 Opensuse-1506-amd64-base llama-server[15167]:         Require all granted
aug 13 21:34:30 Opensuse-1506-amd64-base systemd-coredump[15201]: [🡕] Process 15167 (llama-server) of user 1000 dumped core.

Different ending of the input than what's passed along as input argument to the regex function, which ends with

Then Apache will follow that and forward request to PHP-FPM accordingly. That could be done by having CGI script output \"Location: /some.php\" with status code 302, then client requests again; but that's not efficient.\n\nAlternatively, we can propose using 'mod_proxy' with ProxyPassMatch for .php and also use mod_rewrite's [P] flag to", pos=19, as_match=false) at /home/developer/llama.cpp/source/common/regex-partial

python warnings that I saw while running the gdb dump command above

Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'gdb.error'>: There is no field named _S_categories_size
Python Exception <class 'OverflowError'>: int too big to convert

[aldehir]

@mhitza thank you for that, I can see the culprit in the logs. I'll look into it.