[New Model]: Kimi-VL-A3B

[JenZhao]

The model to consider.

https://huggingface.co/collections/moonshotai/kimi-vl-a3b-67f67b6ac91d3b03d382dd85

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[WangErXiao]

Looking forward to this feature！🚀

[zhouzaida]

Thank you all for your attention. We will propose a MR today. github repo

[courage17340]

Our PR is here: #16387

[tao-xiaoxin]

Our PR is here: #16387

模型下载到本地后启动报错，启动命令如下：

CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
    --port 3000 \
    --served-model-name kimi-vl \
    --trust-remote-code \
    --model /models/Kimi-VL-A3B-Thinking/Kimi-VL-A3B-Thinking \
    --tensor-parallel-size 1 \
    --max-num-batched-tokens 131072 \
    --max-model-len 131072 \
    --max-num-seqs 512 \
    --limit-mm-per-prompt image=256 \
    --disable-mm-preprocessor-cache

报错如下：

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1121, in <module>
    uvloop.run(run_server(args))
  File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
  File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1169, in create_engine_config
    model_config = self.create_model_config()
  File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1057, in create_model_config
    return ModelConfig(
  File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 413, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 486, in _init_multimodal_config
    raise ValueError("`limit_mm_per_prompt` is only supported for "
ValueError: `limit_mm_per_prompt` is only supported for multimodal models.

vllm 版本号：0.8.4 ，请问大佬这个如何解决呢？怎么启动呢？

[courage17340]

Our PR is here: #16387

模型下载到本地后启动报错，启动命令如下：

CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server
--port 3000
--served-model-name kimi-vl
--trust-remote-code
--model /models/Kimi-VL-A3B-Thinking/Kimi-VL-A3B-Thinking
--tensor-parallel-size 1
--max-num-batched-tokens 131072
--max-model-len 131072
--max-num-seqs 512
--limit-mm-per-prompt image=256
--disable-mm-preprocessor-cache
报错如下：

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1121, in <module>
    uvloop.run(run_server(args))
  File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
  File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1169, in create_engine_config
    model_config = self.create_model_config()
  File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1057, in create_model_config
    return ModelConfig(
  File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 413, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 486, in _init_multimodal_config
    raise ValueError("`limit_mm_per_prompt` is only supported for "
ValueError: `limit_mm_per_prompt` is only supported for multimodal models.

vllm 版本号：0.8.4 ，请问大佬这个如何解决呢？怎么启动呢？

看起来像是你的vllm版本里面还没有包含这个PR。这个PR是在0.8.4发版后才合入主线的，可以考虑build from source试试，或者等待下一个vllm版本

[tao-xiaoxin]

分享下我成功运行的例子,首先安装依赖包:

python -m pip install --upgrade pip
pip install transformers wheel setuptools
pip install torch torchvision torchaudio
pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git"
pip install flash-attn --no-build-isolation

最后使用vllm 启动kim-vl 模型:

python -m vllm.entrypoints.openai.api_server \
    --port 8888 \
    --served-model-name kimi-vl \
    --trust-remote-code \
    --model moonshotai/Kimi-VL-A3B-Thinking \
    --tensor-parallel-size 1 \
    --max-num-batched-tokens 131072 \
    --max-model-len 131072 \
    --max-num-seqs 512 \
    --limit-mm-per-prompt image=256 \
    --disable-mm-preprocessor-cache \
    --gpu_memory_utilization 0.9

[nigthDust]

#16358

I attempted this. However, for the step of "pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git"", due to the network restriction of my current environment, I had to clone the main branch from GitHub on another computer using "git clone" command, and then proceed with "pip install ." on that
Encountered some problems

[DarkLight1337]

Closing as completed by #16387

If you have any problems running the model, please open a new issue.