[Bug]: mm_cache keyerror #16903
Description
Your current environment
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \ --model deepseek-ai/deepseek-vl2 \ --hf_overrides '{"architectures":["DeepseekVLV2ForCausalLM"]}' \ --port 6006 \ --dtype=bfloat16 \ --chat_template chat_template.jinja
ERROR 04-19 15:50:28 [core.py:387] EngineCore hit an exception: Traceback (most recent call last):
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/cachetools/__init__.py", line 68, in __getitem__
ERROR 04-19 15:50:28 [core.py:387] return self.__data[key]
ERROR 04-19 15:50:28 [core.py:387] ~~~~~~~~~~~^^^^^
ERROR 04-19 15:50:28 [core.py:387] KeyError: 'a23ac4af3082cb762542f11e83b7d3f6690912dc3b1b32f0c221aea278569f02'
ERROR 04-19 15:50:28 [core.py:387]
ERROR 04-19 15:50:28 [core.py:387] During handling of the above exception, another exception occurred:
ERROR 04-19 15:50:28 [core.py:387]
ERROR 04-19 15:50:28 [core.py:387] Traceback (most recent call last):
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 380, in run_engine_core
ERROR 04-19 15:50:28 [core.py:387] engine_core.run_busy_loop()
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 400, in run_busy_loop
ERROR 04-19 15:50:28 [core.py:387] self._process_input_queue()
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 425, in _process_input_queue
ERROR 04-19 15:50:28 [core.py:387] self._handle_client_request(*req)
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 441, in _handle_client_request
ERROR 04-19 15:50:28 [core.py:387] self.add_request(request)
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 177, in add_request
ERROR 04-19 15:50:28 [core.py:387] request.mm_inputs = self.mm_input_cache_server.get_and_update_p1(
ERROR 04-19 15:50:28 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/vllm/v1/engine/mm_input_cache.py", line 76, in get_and_update_p1
ERROR 04-19 15:50:28 [core.py:387] mm_input = self.mm_cache[mm_hash]
ERROR 04-19 15:50:28 [core.py:387] ~~~~~~~~~~~~~^^^^^^^^^
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/cachetools/__init__.py", line 211, in __getitem__
ERROR 04-19 15:50:28 [core.py:387] value = cache_getitem(self, key)
ERROR 04-19 15:50:28 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/cachetools/__init__.py", line 70, in __getitem__
ERROR 04-19 15:50:28 [core.py:387] return self.__missing__(key)
ERROR 04-19 15:50:28 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^
ERROR 04-19 15:50:28 [core.py:387] File "/root/miniconda3/lib/python3.12/site-packages/cachetools/__init__.py", line 97, in __missing__
ERROR 04-19 15:50:28 [core.py:387] raise KeyError(key)
ERROR 04-19 15:50:28 [core.py:387] KeyError: 'a23ac4af3082cb762542f11e83b7d3f6690912dc3b1b32f0c221aea278569f02'
ERROR 04-19 15:50:28 [core.py:387]
CRITICAL 04-19 15:50:28 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
🐛 Describe the bug
vllm 0.8.4
transformers 4.51.3
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.