Core Quantization Format Conversion Test Coverage in COG-GTM/llama.cpp (AT-103)

[devin-ai-integration]

Summary

This PR implements comprehensive test coverage for quantization format conversions and cross-format accuracy validation as specified in ticket AT-103.

Link to Devin run: https://app.devin.ai/sessions/a58973415a4e4bca823d567a8431b749
Requested by: Alex Peng ([email protected]) / @alexpeng-cognition

Changes

New Test Suites

1. tests/test-conversion-accuracy.cpp - Dedicated test suite for conversion pipeline accuracy

   • Single format quantization/dequantization tests
   • Cross-format conversion tests (e.g., FP16 → Q4_0 → Q8_0 → FP32)
   • Round-trip conversion tests (format A → format B → format A)
   • Tensor alignment validation
   • Large model simulation with memory constraints
   • Multi-file model support testing

2. gguf-py/gguf/conversion_validation.py - Python utilities for HuggingFace to GGUF conversion validation

   • RMSE and maximum error calculations
   • Configurable error thresholds per quantization type
   • Tensor-level and model-level validation
   • JSON report generation

Extended Existing Tests

3. tests/test-backend-ops.cpp - Added test_quant_conversion struct

   • Systematic cross-format conversion tests
   • Tests all major quantization format pairs (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q4_K, Q5_K, Q6_K)

4. tests/test-quantize-fns.cpp - Added cross-format validation functions

   • cross_format_conversion_error() - Tests conversion accuracy between two formats
   • round_trip_error() - Tests quantization stability through round-trip conversions
   • Automated test sections with configurable error thresholds

5. tests/test-quantize-stats.cpp - Added perplexity measurement framework

   • calculate_perplexity() - Quality assessment via perplexity calculation
   • compare_perplexity_across_formats() - Framework for systematic comparison

6. tests/CMakeLists.txt - Added test-conversion-accuracy target

Test Coverage

All quantization formats are tested systematically:

• Base formats: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1
• K-quants: Q2_K, Q3_K, Q4_K, Q5_K, Q6_K
• IQ variants: IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ1_S, IQ1_M, IQ4_NL, IQ3_S, IQ4_XS

Error Thresholds

Error thresholds are based on quantization bit depth following existing patterns from test-quantize-fns.cpp:

• Standard quantization: 0.002 RMSE
• 2-bit quantization: 0.0075 RMSE
• 3-bit quantization: 0.004 RMSE
• Cross-format conversions: 0.01 RMSE (more lenient for multi-hop)
• Round-trip conversions: 0.015 RMSE

Testing

All tests compile successfully and execute:

cmake --build build --target test-conversion-accuracy
cmake --build build --target test-quantize-fns
cmake --build build --target test-backend-ops

./build/bin/test-conversion-accuracy
./build/bin/test-quantize-fns

Backward Compatibility

• All changes extend existing infrastructure without breaking compatibility
• Existing test patterns and error thresholds preserved
• No modifications to production code, only test infrastructure

Related Ticket

Ticket AT-103: Implement comprehensive test coverage for quantization format conversions and cross-format accuracy validation

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring