[question] Does llm-compressor support K/V cache quantization?

[liye0626]

Thanks to llm-compressor for its help in model compression. I would like to ask some K/V cache quantization questions about llm-compressor:

• Does llm-compressor currently support K/V cache quantization? It seems that this is not supported currently. If yes, will llm-compressor support K/V cache quantization in the future?
• vLLM seems to support K/V cache quantization. Is K/V cache quantization supported only in vLLM and not in llm-compressor?

[brian-dellabetta]

Hi @liye0626 , check out https://github.com/vllm-project/llm-compressor/tree/main/examples/quantization_kv_cache for a README and a few examples of kv-cache quantization.

[MichoChan]

can support gemma3？

[brian-dellabetta]

Hi @MichoChan , I don't think we've specifically validated gemma3 with kv-cache qunatization, but you should be able to do so by merging these two examples:

• https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_kv_cache/gemma2_fp8_kv_example.py
• https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/gemma3_example.py

if you encounter an issue please let us know. if it requires any additional code we can add it as an example, user contributions welcome!

[MichoChan]

Hi @MichoChan , I don't think we've specifically validated gemma3 with kv-cache qunatization, but you should be able to do so by merging these two examples:

• https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_kv_cache/gemma2_fp8_kv_example.py
• https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/gemma3_example.py

if you encounter an issue please let us know. if it requires any additional code we can add it as an example, user contributions welcome!

i just install llmcompressor 0.6.0.1 and run https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_kv_cache/gemma2_fp8_kv_example.py, but run with error:
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/entrypoints/oneshot.py", line 149, in call
self.apply_recipe_modifiers(
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/entrypoints/oneshot.py", line 193, in apply_recipe_modifiers
pipeline(self.model, calibration_dataloader, self.dataset_args)
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/pipelines/independent/pipeline.py", line 49, in call
pipeline(model, dataloader, dataset_args)
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/pipelines/sequential/pipeline.py", line 66, in call
subgraphs = trace_subgraphs(model, sample_input, sequential_targets, ignore)
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/pipelines/sequential/helpers.py", line 121, in trace_subgraphs
stack.enter_context(autowrap_forwards(ancestors, ignore))
File "/usr/lib/python3.10/contextlib.py", line 492, in enter_context
result = _cm_type.enter(cm)
File "/usr/lib/python3.10/contextlib.py", line 135, in enter
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/pipelines/sequential/ast_helpers.py", line 31, in autowrap_forwards
stack.enter_context(autowrap_forward(module, ignore))
File "/usr/lib/python3.10/contextlib.py", line 492, in enter_context
result = _cm_type.enter(cm)
File "/usr/lib/python3.10/contextlib.py", line 135, in enter
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/llmcompressor/pipelines/sequential/ast_helpers.py", line 79, in autowrap_forward
new_forward = namespace["forward"].get(module)
KeyError: 'forward'

[liye0626]

Hello @brian-dellabetta @MichoChan , may I ask if w8a8 int8 quantization can be merged with kv cache quantization? - https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_int8/llama3_example.py - https://github.com/vllm-project/llm-compressor/tree/main/examples/quantization_kv_cache The format of the recipe in these two guides is different. The recipe of "quantization_kv_cache" is the str and that of "quantization_w8a8_int8" is the modifier list

…

---- Replied Message ---- | From | Brian ***@***.***> | | Date | 08/08/2025 03:32 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [vllm-project/llm-compressor] [question] Does llm-compressor support K/V cache quantization? (Issue #1711) | brian-dellabetta left a comment (vllm-project/llm-compressor#1711) Hi @liye0626 , check out https://github.com/vllm-project/llm-compressor/tree/main/examples/quantization_kv_cache for a README and a few examples of kv-cache quantization. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[liye0626]

Hello @brian-dellabetta @MichoChan , may I ask these questions:

• do vllm only support FP8 quantization for KV cache quantization?
• Is FP8 KV cache quantization coupled with weight activation FP8 quantization?

[brian-dellabetta]

Hi @MichoChan , my mistake, I told you wrong about Gemma3. It looks like the issue with Gemma3 is known and a workaround is in progress:

• [Autowrapper] Support Gemma3, autowrapper improvements #1693

[brian-dellabetta]

Hi @liye0626 , you can use this as your recipe. It will do W8A8 (floating point) and fp8 kv-cache quantization

"""
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            ignore: ["lm_head"]
            config_groups:
                group_0:
                    weights:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    input_activations:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    targets: ["Linear"]
            kv_cache_scheme:
                num_bits: 8
                type: float
                strategy: tensor
                dynamic: false
                symmetric: true
"""

More generally, kv-cache quantization will be updated and enhanced later this year. Right now it is pretty limited in its capabilities. Shorter term, there is a community PR to allow int8 per-channel quantization:

• [KV Cache] support kv cache int8 per channel quantization #1663

[liye0626]

Hi @liye0626 , you can use this as your recipe. It will do W8A8 (floating point) and fp8 kv-cache quantization

"""
quant_stage:
quant_modifiers:
QuantizationModifier:
ignore: ["lm_head"]
config_groups:
group_0:
weights:
num_bits: 8
type: float
strategy: tensor
dynamic: false
symmetric: true
input_activations:
num_bits: 8
type: float
strategy: tensor
dynamic: false
symmetric: true
targets: ["Linear"]
kv_cache_scheme:
num_bits: 8
type: float
strategy: tensor
dynamic: false
symmetric: true
"""
More generally, kv-cache quantization will be updated and enhanced later this year. Right now it is pretty limited in its capabilities. Shorter term, there is a community PR to allow int8 per-channel quantization:

• [KV Cache] support kv cache int8 per channel quantization #1663

Hi, thank you for your reply. When using vllm to evaluate KV cache quantization, is lm_eval the only option?

MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-KV
lm_eval \

--model vllm \

--model_args pretrained=$MODEL,kv_cache_dtype=fp8,add_bos_token=True \

--tasks gsm8k --num_fewshot 5 --batch_size auto

I mean, can I do a simple test using the following code:

from vllm import LLM
model = LLM("TinyLlama-1.1B-Chat-v1.0-INT8")
output = model.generate("My name is")

When using lm_eval, we can specify "kv_cache_dtype=fp8". How do we set kv_cache_dtype=fp8 for a simple test?

[brian-dellabetta]

Hi @liye0626 , please look into the vllm docs before elevating to us -- https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html?h=kv+cache#example-usage

[liye0626]

Hi @liye0626 , please look into the vllm docs before elevating to us -- https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html?h=kv+cache#example-usage

Thank you very much for your reply.

The config file in Meta-Llama-3-8B-Instruct-FP8-KV (llm-compressor) contains information about the FP8 kv cache quantization. However, when initializing the model in vllm-project/vllm, we have to set the dtype of the fp8 kv cache in LLM.init().