UPSTREAM PR #14891: imatrix: calculate activation-based statistics for new format (GGUF) imatrices

[DajanaV]

Mirrored from ggml-org/llama.cpp#14891

Following up from #9400 and #12718, I've started tinkering with activation-based statistics, in addition to what's currently available via --show-statistics.

At the moment, I'm exploring three options going from from easy to implement and OK approximation, to some assembly required but fairly accurate:

1. L2 norm of activation difference: where larger values would suggest the tensor has significantly transformed the input with respect to the previous layer.
2. KL Divergence reduction using a pre-computed logit file: using a similar approach as described by nostalgebraist in logit lens, and based on a pre-computed logit file (e.g. from a previous llama-perplexity --save-all-logits run)
3. Given that llama-imatrix already generates the actual logits to compute PPL, use Thông T. Nguyễn's logit prism approach to calculate the exact contribution of each layer to the final logit scores

Sharing with the readers, and in particular @compilade and @jukofyork, in case anyone's willing to double check assumptions and/or suggest alternative approaches I haven't considered.

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: PR #33 Impact on LLaMA.cpp

Critical Function Performance Analysis

Core Inference Functions - No Performance Impact

Primary Inference Pipeline:

• llama_decode(): 49,003,788 ns (no change) - Core token processing function
• llama_tokenize(): 834,830 ns (no change) - Text-to-token conversion
• llama_model_load_from_file(): 333,127,900 ns (no change) - Model loading
• llama_batch_init(): 257 ns (no change) - Batch initialization

Key Finding: All critical inference functions show zero performance degradation, indicating the imatrix statistical enhancements do not impact core inference performance.

Affected Functions

Standard Library Components:

• _RegexMask constructor: +0.082% Response Time (+0.018 ns)
• _Construct template: +0.109% Bottleneck (+0.011 ns)

These functions are part of C++ standard library components used in grammar processing, not core inference paths.

Key Performance Indicators Impact Analysis

1. Tokens Per Second - No Impact

Analysis: Core tokenization and inference functions show no performance changes.

Critical Functions Status:

• llama_decode(): 0% change (49,003,788 ns)
• llama_tokenize(): 0% change (834,830 ns)
• llama_encode(): Not measured but no code modifications

Inference: Based on the reference that 2ms slower llama_decode() results in 7% fewer tokens per second, the zero change in llama_decode() indicates no impact on tokens per second performance.

2. Power Consumption - Negligible Impact

Binary-Level Analysis:

• build.bin.libllama.so: <0.001% increase (306,978.34 nJ vs 306,978.33 nJ)
• build.bin.libggml.so: 0% change (6,339.24 nJ)
• build.bin.libggml-cpu.so: 0% change (151,692.17 nJ)
• build.bin.libggml-base.so: 0% change (90,434.19 nJ)

Impact: Minimal power consumption increase limited to the main library binary.

3. Quantization Efficiency - No Direct Impact

Analysis: Quantization-related functions show no performance changes.

Critical Functions Status:

• llama_model_quantize(): Not directly measured but no code modifications in quantization paths
• Model loading functions: No performance impact

Note: The imatrix enhancements improve quantization quality through better statistical analysis but do not affect quantization performance.

4. Memory Usage - Potential Increase

Code Analysis Findings:

• New data structures: Added activations vector to Stats struct
• Memory doubling: Each tensor now stores both activations and squared values
• Conditional allocation: Memory increase only when activation_statistics = true

Impact: Memory usage increases when activation statistics are enabled, but no impact on core inference memory patterns.

5. Batch Processing - No Impact

Critical Functions Status:

• llama_batch_init(): 0% change (257 ns)
• llama_decode() with batches: 0% change
• Batch allocation functions: No code modifications

Analysis: Batch processing performance remains unchanged.

Action Items for Performance Optimization

Code-Level Optimizations

Memory Management:

• Implement conditional compilation for activation statistics to eliminate overhead when disabled
• Use memory pools for statistical computations to reduce allocation overhead
• Optimize data structure alignment to minimize cache misses

Template Optimization:

• Provide explicit template specializations for commonly used statistical functions
• Move template implementations to source files to reduce compilation dependencies
• Use SIMD instructions for statistical calculations

Build System Improvements

Compilation Optimization:

• Enable link-time optimization (LTO) to recover performance from increased code complexity
• Use profile-guided optimization (PGO) for statistical computation paths
• Implement conditional compilation flags for statistical features

Binary Size Management:

• Separate statistical functionality into optional shared libraries
• Use function attribute optimization for hot paths
• Implement dead code elimination for unused statistical features

Summary

PR #33 successfully implements enhanced imatrix statistical capabilities with minimal performance impact on core inference functions. The negligible degradation in standard library components (0.082-0.109%) does not affect primary inference performance metrics. The changes primarily impact memory usage when statistical features are enabled, with no measurable effect on tokens per second, quantization performance, or batch processing efficiency.