UPSTREAM PR #16934: chat: Allow reasoning_content to be passed back

[DajanaV]

Mirrored from ggml-org/llama.cpp#16934

This makes it possible for reasoning_content to be passed back to llama-server, which is useful for LLMs like GPT-OSS or Minimax-M2 that were trained for this.

TBH I'm not sure this is the correct approach as I'm not familiar with the code. I've simply made the necessary changes for llama.cpp no longer error out when receiving reasoning_content back from the client.

I've been using GPT-OSS 120B locally with a codex fork that sends reasoning_content back, and it seems to work quite well.

It also requires a slightly modified jinja chat template that replaces "thinking" with "reasoning_content".

If this is the way to go and is merged, I will follow up with a codex PR that makes this configurable so that codex can be used correctly with llama-server.

I've also looked at Minimax M2's chat template and it seems to use reasoning_content to render <think> blocks, which is compatible to how it is done here.

In case someone wants to try my codex fork with this, here's the config you can drop to ~/.codex/config.toml:

profile = "llama_server"

[model_providers.llama_server]
name = "llama-server"
base_url = "http://localhost:8080/v1"
query_params = {"reasoning_effort" = "high"} # doesn't seem like this is currently working, still need to debug

[profiles.oss]
model_provider = "llama_server"
model = "gpt-oss-120b"

This is the llama-server command I use (adjust for what your hardware can handle):

llama-server --no-mmap --no-warmup --model gpt-oss-120b-mxfp4-00001-of-00003.gguf
 -a gpt-oss-120b --ctx-size 524280 -np 4 --jinja -fa on --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 --swa-full --host 0.0.0.0 --chat-template-kwargs '{"reasoning_effort":"low"}' --chat-template-file gptoss.j2

cc @pwilkin

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Critical Functions

Based on the performance data analysis, the observed degradations are minimal and do not affect the core inference pipeline or critical performance functions.

Critical Function Performance Status

Core Inference Functions - No Impact

• llama_decode() - No performance changes detected
• llama_encode() - No performance changes detected
• llama_tokenize() - No performance changes detected
• llama_model_load_from_file() - No performance changes detected
• llama_batch_init() - No performance changes detected

Affected Functions - Non-Critical

• _RegexMask constructor: +0.082% response time (+0.018 ns)
• make_unique<llm_graph_input_pos_bucket>: +0.117% throughput (+0.122 ns)

KPI Impact Analysis

1. Tokens Per Second - No Impact

Status: No degradation in inference performance

• Core tokenization functions (llama_tokenize, llama_detokenize) show no performance changes
• Primary inference function (llama_decode) maintains baseline performance
• Batch processing functions remain unaffected

Reference Impact: With the baseline that 2ms slower llama_decode reduces tokens/second by 7% on the test configuration, the observed changes would have zero impact on tokens per second as no inference-critical functions show measurable degradation.

2. Power Consumption - Negligible Binary Impact

Affected Binaries:

• build.bin.libllama.so: +0.8 nJ (+0.0003% increase)
• build.bin.libggml-base.so: No change
• build.bin.libggml-cpu.so: No change
• build.bin.libggml.so: No change

Analysis: The power consumption increase is within measurement noise and does not affect core computation binaries.

3. Quantization Efficiency - No Impact

Status: No changes detected

• llama_model_quantize() function shows no performance degradation
• Quantization support functions maintain baseline performance
• GGML quantization operations remain unaffected

4. Memory Usage - No Impact

Status: Memory management functions unaffected

• KV cache operations (llama_memory_clear, llama_memory_seq_rm) show no changes
• Memory allocation functions (ggml_gallocr_new, ggml_tallocr_alloc) maintain performance
• Batch memory management remains efficient

5. Batch Processing - No Impact

Status: Batch processing pipeline unaffected

• llama_batch_init(), llama_batch_get_one() show no performance changes
• Dynamic batching logic remains efficient
• Parallel token processing maintains baseline performance

Root Cause Analysis

Grammar Processing Degradation

The observed degradations affect auxiliary systems rather than core inference:

• Template instantiation overhead: make_unique specialization shows minor compiler optimization differences
• Regex initialization: _RegexMask constructor experiences minimal initialization cost increase
• Compiler optimization variance: Changes likely reflect minor differences in instruction scheduling rather than algorithmic issues

Action Items

Immediate Code Optimizations

1. Template Specialization Review

   • Examine llm_graph_input_pos_bucket constructor for potential pre-allocation opportunities
   • Consider compile-time initialization for frequently used graph components

2. Memory Allocation Patterns

   • Review memory allocation patterns in graph optimization pipeline
   • Evaluate if position buckets can be pre-allocated during graph construction

Build System Optimizations

1. Compiler Flag Verification

   • Ensure consistent optimization flags across template instantiations
   • Verify profile-guided optimization (PGO) coverage for affected functions

2. Template Optimization

   • Consider explicit template instantiation for frequently used specializations
   • Evaluate template parameter optimization for graph components

Performance Impact Assessment

Overall System Impact: Minimal

• Core inference pipeline maintains full performance
• Auxiliary system degradations are within acceptable variance
• No impact on primary LLaMA.cpp performance metrics

Monitoring Focus:

• Track template instantiation performance in future builds
• Monitor for consistency in compiler optimization across releases

The analysis confirms that LLaMA.cpp's critical performance functions remain unaffected, with observed degradations limited to non-essential auxiliary systems that do not impact inference throughput, power efficiency, or memory utilization.

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Project

Critical Function Performance Changes

Based on the analysis of version 60af0985-b86f-4455-89da-7b1a290af1e6 compared to base version 6a6ce734-560c-4bf5-8d96-8f68e7716a0f, the performance degradations are concentrated in C++ standard library functions rather than core LLaMA.cpp inference functions.

Functions with Performance Degradation

Response Time: std::__codecvt_abstract_base<wchar_t, char, __mbstate_t>::in()

• Change: +0.068% (+0.02 ns, from 29.41 ns to 29.43 ns)
• Location: Standard library codecvt implementation
• Impact: Character encoding conversion operations

Throughput: std::__detail::_Scanner<wchar_t>::~_Scanner()

• Change: +0.079% (+0.01 ns, from 18.88 ns to 18.89 ns)
• Location: C++ regex scanner destructor
• Impact: Regular expression cleanup operations

Bottleneck: std::__detail::_Scanner<char>::~_Scanner()

• Change: +0.105% (+0.02 ns, from 14.32 ns to 14.33 ns)
• Location: C++ regex scanner destructor
• Impact: Internal cleanup bottlenecks

Core LLaMA.cpp Functions Status

No Performance Impact Detected in critical inference functions:

• llama_decode() - No changes detected
• llama_encode() - No changes detected
• llama_tokenize() - No changes detected
• llama_model_load_from_file() - No changes detected
• Memory management functions - No changes detected
• Batch processing functions - No changes detected

KPI Impact Analysis

1. Tokens Per Second

Impact: None

• No changes detected in core tokenization/inference functions (llama_decode, llama_encode, llama_tokenize)
• The observed degradations are in standard library utility functions unrelated to token processing
• Expected tokens per second performance remains unchanged

2. Power Consumption

Impact: Negligible

• build.bin.libllama.so: -0.0% change (306,893.80 nJ vs 306,894.13 nJ base)
• build.bin.libggml-base.so: 0.0% change
• build.bin.libggml-cpu.so: 0.0% change
• build.bin.libggml.so: 0.0% change
• Overall power efficiency remains stable across all binaries

3. Quantization Efficiency

Impact: None

• No changes detected in quantization-related functions
• llama_model_quantize() performance unchanged
• Quantization format handling unaffected

4. Memory Usage

Impact: None

• Memory management functions show no performance changes
• KV cache operations (llama_memory_clear, llama_memory_seq_rm) unaffected
• GGML memory allocation functions unchanged

5. Batch Processing

Impact: None

• Batch processing functions show no performance degradation
• llama_batch_init(), llama_batch_get_one() performance unchanged
• Parallel token processing efficiency maintained

Root Cause Analysis

Standard Library Function Degradations

The performance changes stem from C++ standard library functions used for:

• Character encoding conversion (codecvt functions)
• Regular expression processing (regex scanner destructors)

Control Flow Analysis

CFG analysis reveals identical assembly code between versions for the affected functions, indicating the degradation sources are:

• Microarchitectural factors: Instruction cache alignment changes
• Memory layout differences: Function placement affecting cache locality
• System-level variations: CPU frequency scaling or memory subsystem timing

Action Items

Immediate Actions

1. Code Layout Investigation

   • Analyze object file layout changes using objdump to identify function placement differences
   • Compare instruction cache miss rates between versions using performance counters

2. Build Configuration Review

   • Verify compiler optimization flags consistency between builds
   • Check for differences in link-time optimization settings

Code-Specific Optimizations

1. Standard Library Usage Optimization

   • Review usage patterns of codecvt functions in text processing paths
   • Consider alternative character conversion methods if codecvt is performance-critical

2. Regex Usage Analysis

   • Audit regex scanner usage in parsing operations
   • Evaluate opportunities to replace regex with more efficient string processing

Build System Improvements

1. Reproducible Builds

   • Implement deterministic build ordering to ensure consistent function layout
   • Add build flags for consistent code generation across environments

2. Performance Regression Detection

   • Integrate micro-benchmarks for standard library function performance
   • Add automated performance comparison in CI pipeline

Conclusion

The observed performance changes are minimal and concentrated in standard library utility functions rather than core LLaMA.cpp inference operations. The primary inference KPIs (tokens per second, quantization efficiency, memory usage, batch processing) remain unaffected. The negligible power consumption changes confirm that computational workload has not increased meaningfully.

The identical assembly code analysis confirms these are environmental performance variations rather than algorithmic regressions, making them suitable for build system and deployment environment optimizations rather than code logic changes.