feat(llm-katan): add CPU quantization for faster inference (#556)

* feat(llm-katan): add CPU quantization for faster inference

Add int8 dynamic quantization support for CPU inference to improve
performance of llm-katan in testing scenarios.

Changes:
- Add --quantize/--no-quantize CLI flag (enabled by default)
- Implement int8 quantization in TransformersBackend for CPU
- Gracefully fallback on platforms without quantization support
- Add comprehensive documentation in README

Performance improvements:
- 2-4x faster inference on supported platforms (Linux x86_64)
- 4x memory reduction with quantization
- Minimal quality impact (acceptable for testing)

Platform notes:
- Works best on Linux with x86_64 CPUs
- Gracefully falls back on unsupported platforms (e.g., Mac)
- Users can disable with --no-quantize for full precision

Closes: #552

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Yossi Ovadia <[email protected]>

* fix(llm-katan): apply black formatting with line-length=88

Apply root project black configuration (line-length=88) to match CI
formatting requirements.

Signed-off-by: Yossi Ovadia <[email protected]>

---------

Signed-off-by: Yossi Ovadia <[email protected]>
Co-authored-by: Claude <[email protected]>

Author: yossiovadia

e2e-tests/llm-katan/README.md

@@ -74,12 +74,15 @@ Visit [https://huggingface.co/settings/tokens](https://huggingface.co/settings/t
 ### Basic Usage
 
 ```bash
-# Start server with a tiny model
+# Start server with a tiny model (quantization enabled by default for speed)
 llm-katan --model Qwen/Qwen3-0.6B --port 8000
 
 # Start with custom served model name
 llm-katan --model Qwen/Qwen3-0.6B --port 8001 --served-model-name "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
 
+# Disable quantization for higher accuracy (slower)
+llm-katan --model Qwen/Qwen3-0.6B --port 8000 --no-quantize
+
 # With vLLM backend (optional)
 llm-katan --model Qwen/Qwen3-0.6B --port 8000 --backend vllm
 ```
@@ -179,11 +182,51 @@ curl http://127.0.0.1:8000/v1/models
 curl http://127.0.0.1:8000/health
 ```
 
+## CPU Optimization
+
+LLM Katan includes **automatic int8 quantization** for CPU inference, providing significant performance improvements:
+
+### Performance Gains
+
+- **2-4x faster inference** on CPU (on supported platforms)
+- **4x memory reduction**
+- **Enabled by default** for best testing experience
+- **Minimal quality impact** (acceptable for testing scenarios)
+- **Platform support**: Works best on Linux x86_64; may not be available on all platforms (e.g., Mac)
+
+### When to Use Quantization
+
+✅ **Enabled (default)** - Recommended for:
+
+- Fast E2E testing
+- Development environments
+- CI/CD pipelines
+- Resource-constrained environments
+
+❌ **Disabled (--no-quantize)** - Use when you need:
+
+- Maximum accuracy (though tiny models have limited accuracy anyway)
+- Debugging precision-sensitive issues
+- Comparing with full-precision baselines
+
+### Example Performance
+
+```bash
+# Default: Fast with quantization (~50-100s per inference)
+llm-katan --model Qwen/Qwen3-0.6B
+
+# Slower but more accurate (~200s per inference)
+llm-katan --model Qwen/Qwen3-0.6B --no-quantize
+```
+
+> **Note**: Even with quantization, llm-katan is slower than production tools like LM Studio (which uses llama.cpp with extensive optimizations). For production workloads, use vLLM, Ollama, or similar solutions.
+
 ## Use Cases
 
 ### Strengths
 
 - **Fastest time-to-test**: 30 seconds from install to running
+- **Optimized for CPU**: Automatic int8 quantization for 2-4x speedup
 - **Minimal resource footprint**: Designed for tiny models and efficient testing
 - **No GPU required**: Runs on laptops, Macs, and any CPU-only environment
 - **CI/CD integration friendly**: Lightweight and automation-ready
@@ -223,6 +266,7 @@ Optional:
   --max, --max-tokens INTEGER   Maximum tokens to generate (default: 512)
   -t, --temperature FLOAT       Sampling temperature (default: 0.7)
   -d, --device [auto|cpu|cuda]  Device to use (default: auto)
+  --quantize/--no-quantize      Enable int8 quantization for faster CPU inference (default: enabled)
   --log-level [debug|info|warning|error]  Log level (default: INFO)
   --version                     Show version and exit
   --help                        Show help and exit
@@ -234,8 +278,8 @@ Optional:
 # Custom generation settings
 llm-katan --model Qwen/Qwen3-0.6B --max-tokens 1024 --temperature 0.9
 
-# Force specific device
-llm-katan --model Qwen/Qwen3-0.6B --device cpu --log-level debug
+# Force specific device with full precision (no quantization)
+llm-katan --model Qwen/Qwen3-0.6B --device cpu --no-quantize --log-level debug
 
 # Custom host and port
 llm-katan --model Qwen/Qwen3-0.6B --host 127.0.0.1 --port 9000

e2e-tests/llm-katan/llm_katan/cli.py

@@ -90,6 +90,11 @@
     default="INFO",
     help="Log level (default: INFO)",
 )
+@click.option(
+    "--quantize/--no-quantize",
+    default=True,
+    help="Enable int8 quantization for faster CPU inference (default: enabled)",
+)
 @click.version_option(version=__version__, prog_name="LLM Katan")
 def main(
     model: str,
@@ -101,6 +106,7 @@ def main(
     temperature: float,
     device: str,
     log_level: str,
+    quantize: bool,
 ):
     """
     LLM Katan - Lightweight LLM Server for Testing
@@ -133,6 +139,7 @@ def main(
         max_tokens=max_tokens,
         temperature=temperature,
         device=device.lower(),
+        quantize=quantize,
     )
 
     # Print startup information
@@ -141,6 +148,10 @@ def main(
     click.echo(f"   Served as: {config.served_model_name}")
     click.echo(f"   Backend: {config.backend}")
     click.echo(f"   Device: {config.device_auto}")
+    if config.device_auto == "cpu" and config.quantize:
+        click.echo(f"   Quantization: enabled (int8, ~2-4x faster)")
+    elif config.device_auto == "cpu" and not config.quantize:
+        click.echo(f"   Quantization: disabled (full precision)")
     click.echo(f"   Server: http://{config.host}:{config.port}")
     click.echo("")
 

e2e-tests/llm-katan/llm_katan/config.py

@@ -21,6 +21,7 @@ class ServerConfig:
     max_tokens: int = 512
     temperature: float = 0.7
     device: str = "auto"  # "auto", "cpu", "cuda"
+    quantize: bool = True  # Enable int8 quantization for CPU (default: enabled)
 
     def __post_init__(self):
         """Post-initialization processing"""

e2e-tests/llm-katan/llm_katan/model.py

@@ -89,6 +89,31 @@ async def load_model(self) -> None:
         if device == "cpu":
             self.model = self.model.to("cpu")
 
+            # Apply quantization for faster CPU inference (2-4x speedup)
+            if self.config.quantize:
+                logger.info("Applying int8 quantization for CPU optimization...")
+                try:
+                    self.model = torch.quantization.quantize_dynamic(
+                        self.model, {torch.nn.Linear}, dtype=torch.qint8
+                    )
+                    logger.info(
+                        "✓ Quantization applied (2-4x faster inference, 4x less memory)"
+                    )
+                except RuntimeError as e:
+                    if "NoQEngine" in str(e):
+                        logger.warning(
+                            "⚠️  Quantization not supported on this platform - "
+                            "continuing with full precision"
+                        )
+                        logger.info(
+                            "Note: PyTorch quantization requires specific CPU features. "
+                            "Your model will run without quantization."
+                        )
+                    else:
+                        raise
+            else:
+                logger.info("Quantization disabled - using full precision (slower)")
+
         logger.info(f"Model loaded successfully on {device}")
 
     async def generate(

e2e-tests/llm-katan/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "llm-katan"
-version = "0.1.9"
+version = "0.1.10"
 description = "LLM Katan - Lightweight LLM Server for Testing - Real tiny models with FastAPI and HuggingFace"
 readme = "README.md"
 authors = [