UPSTREAM PR #15550: quantize: add option to automatically choose optimal quant types to reach a bpw target at lowest error

[DajanaV]

Mirrored from ggml-org/llama.cpp#15550

This PR introduces a new option --target-bpw implementing an optimised quant type selection algorithm to automatically determine per-tensor quantisation types in order to achieve a target bits-per-weight (bpw), with minimal estimated quality loss.

The selection algorithm,

• builds a candidate set of quant types (K or IQ types)
• for each layer/tensor, it simulates quantise→dequantise per candidate type, and estimates error using a weighted MSE error function. If the imatrix includes activations, it adds a bias penalty term to better reflect forward‑pass impact, making the error estimation more accurate and thus the quant type selection
• it filters candidates to the pareto frontier (lowest error for a given size), then starts from the smallest bpw mix increasing to larger formats, based on the best error‑reduction per added bit, until the global bpw budget is reached
• returns a map of tensor name → ggml_type overrides, which the main quantisation pass uses. If the minimum achievable BPW already exceeds the target, it returns that minimum.

The target_bpw_type() function will look over all quantisable tensors (e.g. embedding, output, etc.) unless --output-tensor-type, --token-embedding-type, and/or --tensor-type options are also used, in which case they'll take precedence.

--prune-layers can also be used in the same run, in which case the target_bpw_type() will skip the pruned layers and only consider the remaining against the total bpw budget.

Important note:

An imatrix that includes activations is required for the algorithm to work. At the time of writing, this is only available by generating the file using #14891 with the --output-format gguf option.

Typical usage: llama-quantize --imatrix imatrix-with-activations.gguf --target-bpw 5.18 LLM-Model-F16.gguf BPW-Quantized-Q4_K_M.gguf q4_k_m

Special thanks to @ddh0, @AesSedai and @compilade for their contributions during the development of this PR.

PR created in draft until testing is completed

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Critical Functions

Critical Function Performance Status

Core Inference Functions - No Performance Impact

• llama_decode: Response Time: 49,003,776 ns (no change)
• llama_tokenize: Response Time: 834,825 ns (no change)
• llama_model_quantize: Response Time: 6,891,639 ns (no change)
• llama_batch_init: Response Time: 257 ns (no change)
• llama_memory_clear: Response Time: 49 ns (no change)

Affected Functions - Minimal Impact

• __copy_move_b: +0.084 ns response time (+0.042%)
• _M_default_append: +0.128 ns bottleneck (+0.113%)

KPI Impact Analysis

1. Tokens Per Second - No Impact

Status: No degradation in inference performance
Analysis: Core tokenization and inference functions (llama_decode, llama_encode, llama_tokenize) show no measurable performance changes. The reference case of 7% tokens/second reduction from 2ms llama_decode slowdown does not apply here.

Affected Functions: None of the critical inference path functions show performance degradation.

2. Power Consumption - Negligible Impact

Binary-Level Analysis:

• build.bin.libllama.so: -0.298 nJ reduction (-0.0% change)
• build.bin.libggml-base.so: No change (0.0%)
• build.bin.libggml-cpu.so: No change (0.0%)
• build.bin.libggml.so: No change (0.0%)

Total Power: ~556,443 nJ across all binaries with negligible overall change.

3. Quantization Efficiency - Enhanced Capability

Status: Significant improvement in quantization capabilities
New Features:

• Target BPW Algorithm: Automatic quantization type selection for optimal bits-per-weight
• Pareto Optimization: Convex hull algorithm for error-size trade-offs
• Multi-threaded Processing: Parallel tensor analysis with signal handling

Performance Impact: The llama_model_quantize function maintains identical performance (6,891,639 ns) despite substantial algorithmic enhancements.

4. Memory Usage - Increased During Quantization

Status: Additional memory structures for enhanced quantization
New Data Structures:

• Activations Data: std::unordered_map<std::string, std::vector<float>>
• Statistics Data: std::unordered_map<std::string, std::vector<float>>
• BPW State Management: Checkpoint and resume functionality

Runtime Impact: Memory overhead only during quantization process, no impact on inference.

5. Batch Processing - No Impact

Status: Batch processing functions maintain identical performance
Analysis:

• llama_batch_init: 257 ns (no change)
• Batch allocation and management: No performance degradation
• Parallel processing: Enhanced through new multi-threaded quantization algorithm

Root Cause Analysis

Performance Degradation Source

The minimal performance impact (0.084 ns in __copy_move_b) stems from:

1. Code Size Inflation

• 1,487 line additions increase binary size
• Instruction cache pressure affects unrelated functions
• Branch prediction changes due to altered code layout

2. Memory Allocation Patterns
The _M_default_append function shows increased bottleneck time due to:

• Enhanced vector operations for token data structures
• Additional memory allocations for new data containers
• Complex object construction in vocabulary management

Control Flow Analysis

The _M_default_append CFG shows:

• 27 basic blocks with complex exception handling
• Multiple PLT calls for memory allocation (185-279 ns each)
• Exception safety mechanisms adding overhead to vector operations

Action Items

Immediate Optimizations

1. Binary Size Management

• Enable Link-Time Optimization (LTO) to reduce code size impact
• Profile instruction cache behavior during compilation
• Consider function placement optimization for critical paths

2. Memory Allocation Optimization

• Implement memory pool allocation for frequent vector operations
• Pre-allocate vocabulary containers to reduce dynamic allocation
• Optimize token data structure layout for cache efficiency

3. Code Organization

• Modularize the 1,136-line target_bpw_type() function into smaller components
• Extract algorithmic utilities (Pareto optimization, Lagrangian relaxation)
• Separate data structures from processing logic

Build System Enhancements

1. Compilation Optimization

• Profile-Guided Optimization (PGO) for branch prediction improvement
• Function section placement to minimize cache conflicts
• Template instantiation optimization for STL containers

2. Memory Layout Optimization

• Align data structures for vectorized operations
• Optimize token data padding to reduce memory footprint
• Implement custom allocators for performance-critical containers

Conclusion

The changes introduce substantial quantization enhancements with minimal performance impact on core inference functions. The 0.084 ns degradation represents a negligible trade-off for significant algorithmic improvements. The enhanced quantization capabilities provide substantial value while maintaining inference performance integrity.