[New Model]: Kimi-VL models are already supported on vllm, but I cannnot run them directly

[YuanCheng-coder]

The model to consider.

Kimi-VL-A3B-Instruct

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

vllm-project/vllm#16387 here is the suppor PR but I cannot run it directly. Below is the ERROR

(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/kimi_vl.py", line 302, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.language_model = DeepseekV2Model(
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/compilation/decorators.py", line 183, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/deepseek_v2.py", line 674, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.start_layer, self.end_layer, self.layers = make_layers(
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/utils.py", line 640, in make_layers
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] [PPMissingLayer() for _ in range(start_layer)] + [
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/utils.py", line 641, in
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/deepseek_v2.py", line 676, in
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] lambda prefix: DeepseekV2DecoderLayer(
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/deepseek_v2.py", line 562, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.self_attn = attn_cls(
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/model_executor/models/deepseek_v2.py", line 475, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.mla_attn = Attention(
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm/vllm/attention/layer.py", line 175, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads,
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] File "/vllm-workspace/main-vllm-ascend/vllm_ascend/attention/mla_v1.py", line 455, in init
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] self.rotary_emb = kwargs['rotary_emb']
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] ~~~~~~^^^^^^^^^^^^^^
(EngineCore_0 pid=12216) ERROR 08-27 11:28:55 [core.py:700] KeyError: 'rotary_emb'

Version of vllm and vllm-ascend

[Yikun]

@zhangxinyuehfad Please take a look

[zhangxinyuehfad]

I reproduced this issue by the latest code vllm 91e382c and vllm-ascend 936c102

(VllmWorker TP0 pid=3069) INFO 08-28 01:47:33 [parallel_state.py:1134] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker TP1 pid=3070) INFO 08-28 01:47:33 [parallel_state.py:1134] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(VllmWorker TP0 pid=3069) INFO 08-28 01:47:34 [model_runner_v1.py:2261] Starting to load model moonshotai/Kimi-VL-A3B-Instruct...
(VllmWorker TP1 pid=3070) INFO 08-28 01:47:34 [model_runner_v1.py:2261] Starting to load model moonshotai/Kimi-VL-A3B-Instruct...
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565] WorkerProc failed to start.
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565] Traceback (most recent call last):
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 539, in worker_main
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     worker = WorkerProc(*args, **kwargs)
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 408, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.worker.load_model()
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 241, in load_model
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.model_runner.load_model()
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2264, in load_model
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 118, in get_model
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     return loader.load_model(vllm_config=vllm_config,
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 44, in load_model
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     model = initialize_model(vllm_config=vllm_config,
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/utils.py", line 63, in initialize_model
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/kimi_vl.py", line 303, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.language_model = DeepseekV2Model(
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                           ^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 199, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/deepseek_v2.py", line 674, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                                                     ^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 642, in make_layers
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     [PPMissingLayer() for _ in range(start_layer)] + [
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                                                      ^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 643, in <listcomp>
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/deepseek_v2.py", line 676, in <lambda>
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     lambda prefix: DeepseekV2DecoderLayer(
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                    ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/deepseek_v2.py", line 562, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.self_attn = attn_cls(
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                      ^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/model_executor/models/deepseek_v2.py", line 475, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.mla_attn = Attention(
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                     ^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm/vllm/attention/layer.py", line 182, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads,
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]   File "/vllm-workspace/vllm-ascend/vllm_ascend/attention/mla_v1.py", line 452, in __init__
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]     self.rotary_emb = kwargs['rotary_emb']
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565]                       ~~~~~~^^^^^^^^^^^^^^
(VllmWorker TP0 pid=3069) ERROR 08-28 01:47:35 [multiproc_executor.py:565] KeyError: 'rotary_emb'
(VllmWorker TP0 pid=3069) INFO 08-28 01:47:35 [multiproc_executor.py:526] Parent process exited, terminating worker
(VllmWorker TP1 pid=3070) INFO 08-28 01:47:35 [multiproc_executor.py:526] Parent process exited, terminating worker
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711] EngineCore failed to start.
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711] Traceback (most recent call last):
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 702, in run_engine_core
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 503, in __init__
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 80, in __init__
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     self.model_executor = executor_class(vllm_config)
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     self._init_executor()
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 96, in _init_executor
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 478, in wait_for_ready
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711]     raise e from None
(EngineCore_0 pid=2930) ERROR 08-28 01:47:39 [core.py:711] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore_0 pid=2930) Process EngineCore_0:
(EngineCore_0 pid=2930) Traceback (most recent call last):
(EngineCore_0 pid=2930)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_0 pid=2930)     self.run()
(EngineCore_0 pid=2930)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_0 pid=2930)     self._target(*self._args, **self._kwargs)
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 715, in run_engine_core
(EngineCore_0 pid=2930)     raise e
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 702, in run_engine_core
(EngineCore_0 pid=2930)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_0 pid=2930)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 503, in __init__
(EngineCore_0 pid=2930)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 80, in __init__
(EngineCore_0 pid=2930)     self.model_executor = executor_class(vllm_config)
(EngineCore_0 pid=2930)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_0 pid=2930)     self._init_executor()
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 96, in _init_executor
(EngineCore_0 pid=2930)     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore_0 pid=2930)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=2930)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 478, in wait_for_ready
(EngineCore_0 pid=2930)     raise e from None
(EngineCore_0 pid=2930) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(APIServer pid=2660) Traceback (most recent call last):
(APIServer pid=2660)   File "/usr/local/python3.11.13/bin/vllm", line 8, in <module>
(APIServer pid=2660)     sys.exit(main())
(APIServer pid=2660)              ^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=2660)     args.dispatch_function(args)
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 50, in cmd
(APIServer pid=2660)     uvloop.run(run_server(args))
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
(APIServer pid=2660)     return runner.run(wrapper())
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=2660)     return self._loop.run_until_complete(task)
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=2660)     return await main
(APIServer pid=2660)            ^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1930, in run_server
(APIServer pid=2660)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1950, in run_server_worker
(APIServer pid=2660)     async with build_async_engine_client(
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=2660)     return await anext(self.gen)
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client
(APIServer pid=2660)     async with build_async_engine_client_from_engine_args(
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=2660)     return await anext(self.gen)
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 220, in build_async_engine_client_from_engine_args
(APIServer pid=2660)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=2660)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 1581, in inner
(APIServer pid=2660)     return fn(*args, **kwargs)
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 197, in from_vllm_config
(APIServer pid=2660)     return cls(
(APIServer pid=2660)            ^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 123, in __init__
(APIServer pid=2660)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=2660)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=2660)     return AsyncMPClient(*client_args)
(APIServer pid=2660)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 767, in __init__
(APIServer pid=2660)     super().__init__(
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 446, in __init__
(APIServer pid=2660)     with launch_core_engines(vllm_config, executor_class,
(APIServer pid=2660)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=2660)     next(self.gen)
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 706, in launch_core_engines
(APIServer pid=2660)     wait_for_engine_startup(
(APIServer pid=2660)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 759, in wait_for_engine_startup
(APIServer pid=2660)     raise RuntimeError("Engine core initialization failed. "
(APIServer pid=2660) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

[wangxiyuan]

this is because that kimi-vl rely on deepseek model. the deepseek model in vllm-ascend is not the same with the one in vllm. we need either make the deepseek support more generic or rewrite kimi-vl model.

For our long term plan, we'll make the model generic enough. please wait more.

For quick fix, you can replace DeepseekV2Model from vllm to CustomDeepseekV2Model in vllm-ascend.

https://github.com/vllm-project/vllm/blob/5da4f5d857933329aaca779e3a81f1385c84e34a/vllm/model_executor/models/kimi_vl.py#L64

from vllm_ascend.models.models.deepseek_v2 import CustomDeepseekV2Model as DeepseekV2Model

maybe more tiny place should be updated as well, you can test it locally.