vLLM support

[zhouzaida]

We have submitted a Merge Request #16387 to vLLM. You are welcome to deploy Kimi-VL using the branch corresponding to the vLLM MR until the MR is merged.

[loki1017]

Hello！When I use vllm0.8.4 version（Apr 15, 2025） to load the Kimi-VL-Instruct model, the following problem occurs. Hasn't the support for Kimi in vllm already been merged into the main branch?

Error：
ValueError: limit_mm_per_prompt is only supported for multimodal models.

Command：
python3 -m vllm.entrypoints.openai.api_server
--model /moonshotai/Kimi-VL-A3B-Instruct/
--served-model-name kimi-vl
--trust-remote-code
--tensor-parallel-size 1
--max-num-batched-tokens 131072
--max-model-len 131072
--max-num-seqs 512
--limit-mm-per-prompt image=256

Info：
INFO 04-16 17:22:04 [init.py:239] Automatically detected platform cuda.
INFO 04-16 17:22:06 [api_server.py:1034] vLLM API server version 0.8.4
INFO 04-16 17:22:06 [api_server.py:1035] args: Namespace(host='192.168.141.235', port=25414, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/Work/llm_models_common/moonshotai/Kimi-VL-A3B-Instruct/', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config=None, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=131072, guided_decoding_backend='auto', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=None, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=131072, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=512, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 256}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['kimi-vl'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, enable_reasoning=False, reasoning_parser=None, disable_cascade_attn=False, disable_chunked_mm_input=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1121, in
uvloop.run(run_server(args))
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/uvloop/init.py", line 105, in run
return runner.run(wrapper())
^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
async with build_async_engine_client(args) as engine_client:
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 1154, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 1042, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/config.py", line 489, in init
self.multimodal_config = self._init_multimodal_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Miniforge3/envs/vllm0.8.4/lib/python3.11/site-packages/vllm/config.py", line 562, in _init_multimodal_config
raise ValueError("limit_mm_per_prompt is only supported for "
ValueError: limit_mm_per_prompt is only supported for multimodal models.

[zhouzaida]

Hi, Kimi-VL support is not released in vllm v0.8.4, you can use it through the main branch.

[jinzhuoran]

使用最新的vllm会有以下错误：

ERROR 04-21 01:55:57 [core.py:392] EngineCore encountered a fatal error. ERROR 04-21 01:55:57 [core.py:392] Traceback (most recent call last): ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 383, in run_engine_core ERROR 04-21 01:55:57 [core.py:392] engine_core.run_busy_loop() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 407, in run_busy_loop ERROR 04-21 01:55:57 [core.py:392] self._process_engine_step() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 436, in _process_engine_step ERROR 04-21 01:55:57 [core.py:392] outputs = self.step_fn() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 202, in step ERROR 04-21 01:55:57 [core.py:392] output = self.model_executor.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.collective_rpc("execute_model", ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc ERROR 04-21 01:55:57 [core.py:392] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2428, in run_method ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 243, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.model_runner.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1013, in execute_model ERROR 04-21 01:55:57 [core.py:392] self._prepare_inputs(scheduler_output)) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 589, in _prepare_inputs ERROR 04-21 01:55:57 [core.py:392] attn_metadata = self.attn_metadata_builder.build( ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/attention/backends/mla/common.py", line 508, in build ERROR 04-21 01:55:57 [core.py:392] self.page_size) ERROR 04-21 01:55:57 [core.py:392] AttributeError: 'MLACommonMetadataBuilder' object has no attribute 'page_size' Process EngineCore_0:

vllm安装方式为 pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

运行代码 vllm serve /mnt/model/Kimi-VL-A3B-Thinking/ --port 22345 --tensor-parallel-size 1 --limit-mm-per-prompt image=1 --max-model-len 40000 --dtype bfloat16 --trust-remote-code --served-model-name Kimi-VL-A3B-Thinking

[Gaoyg]

使用最新的vllm会有以下错误：

ERROR 04-21 01:55:57 [core.py:392] EngineCore encountered a fatal error. ERROR 04-21 01:55:57 [core.py:392] Traceback (most recent call last): ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 383, in run_engine_core ERROR 04-21 01:55:57 [core.py:392] engine_core.run_busy_loop() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 407, in run_busy_loop ERROR 04-21 01:55:57 [core.py:392] self._process_engine_step() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 436, in _process_engine_step ERROR 04-21 01:55:57 [core.py:392] outputs = self.step_fn() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 202, in step ERROR 04-21 01:55:57 [core.py:392] output = self.model_executor.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.collective_rpc("execute_model", ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc ERROR 04-21 01:55:57 [core.py:392] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2428, in run_method ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 243, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.model_runner.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1013, in execute_model ERROR 04-21 01:55:57 [core.py:392] self._prepare_inputs(scheduler_output)) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 589, in _prepare_inputs ERROR 04-21 01:55:57 [core.py:392] attn_metadata = self.attn_metadata_builder.build( ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/attention/backends/mla/common.py", line 508, in build ERROR 04-21 01:55:57 [core.py:392] self.page_size) ERROR 04-21 01:55:57 [core.py:392] AttributeError: 'MLACommonMetadataBuilder' object has no attribute 'page_size' Process EngineCore_0:

vllm安装方式为 pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

运行代码 vllm serve /mnt/model/Kimi-VL-A3B-Thinking/ --port 22345 --tensor-parallel-size 1 --limit-mm-per-prompt image=1 --max-model-len 40000 --dtype bfloat16 --trust-remote-code --served-model-name Kimi-VL-A3B-Thinking

Same problem

[jinzhuoran]

使用最新的vllm会有以下错误：
ERROR 04-21 01:55:57 [core.py:392] EngineCore encountered a fatal error. ERROR 04-21 01:55:57 [core.py:392] Traceback (most recent call last): ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 383, in run_engine_core ERROR 04-21 01:55:57 [core.py:392] engine_core.run_busy_loop() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 407, in run_busy_loop ERROR 04-21 01:55:57 [core.py:392] self._process_engine_step() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 436, in _process_engine_step ERROR 04-21 01:55:57 [core.py:392] outputs = self.step_fn() ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 202, in step ERROR 04-21 01:55:57 [core.py:392] output = self.model_executor.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 86, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.collective_rpc("execute_model", ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc ERROR 04-21 01:55:57 [core.py:392] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2428, in run_method ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 243, in execute_model ERROR 04-21 01:55:57 [core.py:392] output = self.model_runner.execute_model(scheduler_output) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-21 01:55:57 [core.py:392] return func(*args, **kwargs) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1013, in execute_model ERROR 04-21 01:55:57 [core.py:392] self._prepare_inputs(scheduler_output)) ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 589, in _prepare_inputs ERROR 04-21 01:55:57 [core.py:392] attn_metadata = self.attn_metadata_builder.build( ERROR 04-21 01:55:57 [core.py:392] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/attention/backends/mla/common.py", line 508, in build ERROR 04-21 01:55:57 [core.py:392] self.page_size) ERROR 04-21 01:55:57 [core.py:392] AttributeError: 'MLACommonMetadataBuilder' object has no attribute 'page_size' Process EngineCore_0:
vllm安装方式为 pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
运行代码 vllm serve /mnt/model/Kimi-VL-A3B-Thinking/ --port 22345 --tensor-parallel-size 1 --limit-mm-per-prompt image=1 --max-model-len 40000 --dtype bfloat16 --trust-remote-code --served-model-name Kimi-VL-A3B-Thinking

Same problem

前两天使用pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly 安装，运行是没问题的，估计是因为这两天vllm的更新有问题

[zhouzaida]

可以再重新安装一下试试，我昨天用 main branch 是正常的

[cmatachuan]

pip install vllm --pre --extra-index-url https://wheels.vll

同样遇到了这个问题，最后是怎么解决的呀

[anothersin]

same problem

[zhouzaida]

vllm 0.8.5 版本包含了 kimi-vl 的支持，可以再试试

[jinzhuoran]

vllm 0.8.5 版本包含了 kimi-vl 的支持，可以再试试

仍然无法解决

[SISTMrL]

vllm 0.8.5 版本包含了 kimi-vl 的支持，可以再试试

不行的，第一轮对话可以正常跑，当第二次调用时，就会报错，vllm-project/vllm#16908

[HelloWorldU]

@zhouzaida Hi! When l look at the moonvit.py code implemented in vLLM, l found a place which is strange:

class MoonVitVLProjector(nn.Module):

    def __init__(
        self,
        in_channels: int,
        merge_kernel_size: list[int, int],
        hidden_act: str = "gelu",
        ln_eps: float = 1e-5,
        out_dim: int = 4096,
    ):
        super().__init__()
        self.hidden_size = in_channels * merge_kernel_size[
            0] * merge_kernel_size[1]

        self.pre_norm = nn.nn.LayerNorm(in_channels, eps=ln_eps)
        self.linear_1 = nn.Linear(self.hidden_size,
                                  self.hidden_size,
                                  bias=True)
        self.act = ACT2FN[hidden_act]
        self.linear_2 = nn.Linear(self.hidden_size, out_dim, bias=True)

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.pre_norm(hidden_states).view(-1, self.hidden_size)
        hidden_states = self.linear_1(hidden_states)
        hidden_states = self.act(hidden_states)
        hidden_states = self.linear_2(hidden_states)
        return hidden_states

Does "nn.nn.LayerNorm" expected? It's a little bit confusing.