Cannot set kv cache quantization type when loading a model.

[12bitmisfit]

If I set a config dict like:

config = {
  "contextLength": 131072,
  "offloadKVCacheToGpu": True,
  "llamaKCacheQuantizationType": "q8_0",
  "llamaVCacheQuantizationType": "q8_0",
  "flashAttention": True,
  "gpu": {
    "disabledGpus": [],
    "ratio": 1.0
  },
  "gpuStrictVramCap": True,

  "tryMmap": True
}

and pass the config like:

import lmstudio as lms

with lms.Client() as client:
    client.llm.model("model_key", config=config)

LM Studio will not set kv cache quantization types. It respects other changes like flashAttention and contextLength so I don't think I'm setting the config dict wrong.

[ncoghlan]

Comparing https://github.com/lmstudio-ai/lmstudio-js/blob/2dc7bafbe26d48a1c1d10bdb252e2573b4acf72e/packages/lms-kv-config/src/schema.ts#L274 with

lmstudio-python/src/lmstudio/_kv_config.py

Line 226 in a8f9223

"llamaKCacheQuantizationType": CheckboxField("llamaKCacheQuantizationType"),

this looks to be another case of the independent config mapping implementations in the SDKs causing problems.

lmstudio-ai/lmstudio-js#253 and #32 cover the systematic fix for that problem (deriving the Python config key mapping from the native Typescript one), but the immediate fix here is just to correct the mapping for these keys in the Python SDK.

[ncoghlan]

This has been fixed for 1.4.1:

Loaded via:

>>> import lmstudio as lms
>>> model = lms.llm("llama-3.2-1b-instruct", config = {"llamaKCacheQuantizationType": "q4_0", "llamaVCacheQuantizationType": "f16"})