UPSTREAM PR #16956: Fix garbled output with REPACK at high thread counts

[DajanaV]

Mirrored from ggml-org/llama.cpp#16956

Fixed a race condition in the REPACK matrix multiplication code that caused garbled output when using 26+ threads (model-dependent threshold). The issue occurred because with high thread counts, the code forced chunk count to equal thread count, creating many small chunks. After aligning these chunks to NB_COLS boundaries, adjacent chunks could overlap, causing data corruption and race conditions. The fix enforces minimum chunk sizes based on NB_COLS and caps maximum chunk count to prevent creating too many tiny chunks, ensuring proper alignment without overlaps.

Make sure to read the contributing guidelines before submitting a PR

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Critical Functions

Critical Function Performance Status

Core Inference Functions

All critical inference functions show zero performance degradation:

• llama_decode: 49ms Response Time (0% change) - Primary inference function
• llama_encode: 12ms Response Time (0% change) - Encoder processing
• llama_tokenize: 833μs Response Time (0% change) - Text tokenization
• ggml_backend_graph_compute: 148ns Response Time (0% change) - Core computation

Model Management Functions

Model loading and processing functions maintain stable performance:

• llama_model_load_from_file: 332ms Response Time (0% change)
• llama_model_quantize: 7ms Response Time (0% change)
• llama_batch_init: 257ns Response Time (0% change)

Key Performance Indicators Impact Analysis

1. Tokens Per Second

Status: No Impact

• Critical inference functions unchanged: llama_decode, llama_encode, and llama_tokenize show 0% performance change
• Reference baseline maintained: No degradation in the 49ms llama_decode execution time
• Tokenization pipeline stable: 833μs llama_tokenize performance unchanged

Conclusion: Tokens per second throughput remains unaffected as core inference functions maintain baseline performance.

2. Power Consumption

Status: Negligible Impact

• build.bin.libllama.so: 306,894 nJ (0.0% change)
• build.bin.libggml-cpu.so: 151,692 nJ (0.0% change)
• build.bin.libggml-base.so: 90,434 nJ (0.0% change)
• build.bin.libggml.so: 6,339 nJ (0.0% change)

Conclusion: All binaries maintain stable power consumption profiles with no measurable energy efficiency changes.

3. Quantization Efficiency

Status: No Impact

• llama_model_quantize: 7ms Response Time (0% change)
• Quantization pipeline unchanged: No modifications to quantization algorithms or data paths
• Memory layout preserved: Quantized model structure remains consistent

Conclusion: Quantization performance and efficiency metrics remain at baseline levels.

4. Memory Usage

Status: No Impact

• Memory management functions stable: KV cache and allocation functions show no performance changes
• llama_batch_init: 257ns Response Time (0% change) - Batch memory allocation unchanged
• Memory access patterns preserved: No changes in memory allocation or deallocation timing

Conclusion: Memory usage patterns and allocation efficiency remain unchanged.

5. Batch Processing

Status: No Impact

• llama_batch_init: 257ns Response Time (0% change)
• Batch processing pipeline unchanged: No modifications to batch allocation or processing logic
• Parallel processing efficiency maintained: Core batch functions show stable performance

Conclusion: Batch processing efficiency remains at baseline performance levels.

Action Items for Performance Optimization

Immediate Focus Areas

Based on the analysis, the only measurable performance change is in the _RegexMask constructor (+0.082% increase). However, this function is not part of the critical inference path.

Code-Level Optimizations

1. Template Instantiation Review

   • Examine regex template usage patterns in tokenization modules
   • Consider compile-time regex pattern optimization where applicable

2. Build System Optimization

   • Verify compiler optimization flags for template-heavy code sections
   • Review link-time optimization settings for standard library components

Critical Function Monitoring

Continue monitoring these performance-critical functions for future changes:

• llama_decode (49ms baseline)
• llama_tokenize (833μs baseline)
• llama_encode (12ms baseline)
• ggml_backend_graph_compute (148ns baseline)

Summary

The current version maintains excellent performance stability across all critical functions. No changes impact tokens per second throughput, power consumption, quantization efficiency, memory usage, or batch processing performance. The minor degradation in _RegexMask constructor represents measurement noise rather than functional regression and does not affect inference performance.