[Bug]: kernel error when run "benchmarks/qwen3-omni/vllm_omni/eval_qwen3_moe_omni.sh" #1532
Description
Your current environment
The output of python collect_env.py
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : Could not collect
CUDA_MODULE_LOADING set to :
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3
Nvidia driver version : 535.216.01
cuDNN version : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 208
On-line CPU(s) list: 0-207
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz
CPU family: 6
Model: 143
Thread(s) per core: 2
Core(s) per socket: 52
Socket(s): 2
Stepping: 8
BogoMIPS: 5399.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 4.9 MiB (104 instances)
L1i cache: 3.3 MiB (104 instances)
L2 cache: 208 MiB (104 instances)
L3 cache: 210 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-51,104-155
NUMA node1 CPU(s): 52-103,156-
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.0
[pip3] nvidia-cutlass-dsl-libs-base==4.4.0
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] onnxruntime==1.24.2
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0
[pip3] torchaudio==2.10.0
[pip3] torchsde==0.2.6
[pip3] torchvision==0.25.0
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] numpy 2.1.2 pypi_0 pypi
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.16.0
vLLM-Omni Version : 0.14.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-51,104-155 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-51,104-155 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 52-103,156-207 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 52-103,156-207 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 52-103,156-207 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 52-103,156-207 1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_gcpuser
LD_LIBRARY_PATH=/home/gcpuser/.venv/lib/python3.12/site-packages/cv2/../../lib64:
Your code version
The commit id or version of vllm
vllm-version 0.16.0
The commit id or version of vllm-omni
vllm-omni commit bf9754a8b14b097f777ab7ec8a6ec2d3f80a5334
🐛 Describe the bug
reproduce command
uv run bash vllm-omni/benchmarks/qwen3-omni/vllm_omni/eval_qwen3_moe_omni.sh
error log
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] Traceback (most recent call last):
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 999, in run_engine_core
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] engine_core.run_busy_loop()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1026, in run_busy_loop
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] self._process_engine_step()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1060, in _process_engine_step
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 514, in step_with_batch_queue
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] model_output = future.result()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 81, in result
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] return super().result()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] return self.__get_result()
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] raise self._exception
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 85, in wait_for_response
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] response = self.aggregate(get_response())
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 358, in get_response
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] raise RuntimeError(
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] RuntimeError: Worker failed with error 'CUDA error: device-side assert triggered
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=45035) [Stage-1] ERROR 02-27 03:29:31 [core.py:1008] ', please check the stack trace above for the root cause
-ede7-4e89-9bb1-c869ceea1a4d', '12_41011508-b7d5-43af-abd7-23bfc8982e97', '13_9a310de2-42c9-479d-baef-a24d32db747c', '14_17ea867f-c31d-4b2e-bf87-eb58f5b08e0b', '15_e20280ca-dab7-4107-851a-9d50fad558b4', '16_aa54f891-d56c-46c2-beb4-44a5c1e4949d', '17_9cf216f0-9293-4df2-b08b-cd892a9191b5', '18_c6f91fcd-cba5-476c-b02d-cd88f89725c4', '19_d7236b6d-6335-48eb-bb50-9234138282b2', '20_2497f6cc-d3d4-4a91-9ce5-10081f4bb70e', '21_00496dcd-e257-44af-b8dd-9c78d5bbc0d6', '22_6007150f-e57e-49dd-9da3-fdad92bcd31d', '23_113dedaf-f281-403a-b24a-0f0e05cd725c', '24_a5a2a280-166a-437f-9d02-d57103609435', '25_b56e1df9-e5fe-4da7-8340-989997a7b742', '26_5b8c6384-595a-4bd2-9926-4b17cb76b183', '27_210d5dd6-054f-4971-9ad4-896854b1ff5a', '28_0a104b07-dda0-4cf8-8b51-6e2b2c09cf04', '29_f6e0ca39-6816-4767-83be-b52769451466', '30_c9edf9f5-e54d-439d-8e89-d0ed48afc26c', '31_2b420999-42a3-4bec-a5b0-0b8ef3f167db', '32_dacabaa4-1e30-4c1c-9aa7-10573fae3d9e', '33_a23ab402-be93-4cdb-9dfd-7abf9fb63dda', '34_e718dc11-a88a-403f-87cd-ef81eb7a445c', '35_370240f6-9bb0-4755-ad35-276f4dba0192', '36_01379a49-c81b-4b12-a21b-0efa924c54d3', '37_a4a0cfdd-8c6f-4096-a39e-0372bf67cf96', '38_45bfe0c8-e0dd-46bc-98dc-74e78159a398', '39_51430ac8-9ff9-448d-92b1-2ce8aba6269b', '40_ff87d776-a7ab-49f2-beda-fdf33d332da8', '41_dd6d3d0f-b851-4708-910a-9356595cdcf9', '42_7a97251c-25aa-48b0-9a1a-3977ae689680', '43_a1138186-c34a-435b-8913-021ea897abd1', '44_435cf39e-b62e-4c2c-adc9-454f9e847da2', '45_fab2e8ea-d4d1-4370-b4b5-5366cbc84f5d', '46_07b1df9f-d52f-49d0-b093-5529ccdd6c55', '47_1bcada15-5795-4aae-860e-ce24204c57e3', '48_32863e18-62e6-473b-b18f-7820e765d835', '49_bba9e0da-68d2-4863-ac78-5b13b478bc89', '50_f27d1928-b45d-4a83-9c7a-45873303f689', '51_4248492e-c988-48ef-8d58-302d37e3071d', '52_527971bc-dd35-4c1e-ad40-cec19e907251', '53_a56f7c09-ced8-41d6-a8e3-3ea3570ccc8b', '54_94549a5f-e2f3-4c8e-801c-d9237b6c865d', '55_c5f29b56-b7e1-4936-b16c-5e5f7cc478d5', '56_cb43438c-a87a-4857-9b39-6a6ebb773f51', '57_0accdada-62b3-4911-9f24-abaf00f3ba62', '58_64a29108-57e4-4680-b642-dc7531bd2751', '59_4d41eab2-a9dd-43ca-8d7f-7aeb341e1cdf', '60_9345d03b-1eb0-4c67-b686-744c303d41b2', '61_72315ab2-1cf8-4106-8313-5f47cbac7414', '62_ead5ec60-1c86-420c-b3d4-177e86e745e6', '63_1e1a92ff-f120-4573-aef3-e3733ddc189e']: EngineCore encountered an issue. See stack trace (above) for the root cause.
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] Traceback (most recent call last):
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/vllm-omni/vllm_omni/entrypoints/omni_stage.py", line 968, in _stage_worker
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] results = stage_engine.generate(
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 449, in generate
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] outputs = self._run_completion(
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1760, in _run_completion
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] return self._run_engine(use_tqdm=use_tqdm)
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/vllm-omni/vllm_omni/entrypoints/omni_llm.py", line 217, in _run_engine
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] step_outputs = self.llm_engine.step()
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 294, in step
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] outputs = self.engine_core.get_output()
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 728, in get_output
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] raise self._format_exception(outputs) from None
[Stage-1] ERROR 02-27 03:29:31 [omni_stage.py:1048] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 0_03249bae-0f13-45b0-81c3-3c3e13b9a4d5: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 1_f7865ec7-d8fc-44a8-82e7-7f297d9164c3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 2_36bdb4ee-7255-488f-b670-1442d698134a: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 3_c88af7a3-ace8-4079-b856-3081d21c1c08: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 4_616adc26-6c06-4cfd-838e-a59ab379d697: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 5_58f798fb-4bf3-4d62-9ea8-575e3f0ceecb: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 6_970d13ab-2d61-43bf-958c-85e82f11c1cc: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 7_0287a90d-d653-4571-ba23-3ee42590eec3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 8_e96ea3e0-5091-4cca-8918-c6f256928f8e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 9_2792bfda-5dc6-40f9-bca7-54c0fcd90b9a: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 10_5dbd3e06-9f5b-431a-9a6a-99633b22eef6: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 11_f9af6ba2-ede7-4e89-9bb1-c869ceea1a4d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 12_41011508-b7d5-43af-abd7-23bfc8982e97: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 13_9a310de2-42c9-479d-baef-a24d32db747c: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 14_17ea867f-c31d-4b2e-bf87-eb58f5b08e0b: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 15_e20280ca-dab7-4107-851a-9d50fad558b4: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 16_aa54f891-d56c-46c2-beb4-44a5c1e4949d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 17_9cf216f0-9293-4df2-b08b-cd892a9191b5: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 18_c6f91fcd-cba5-476c-b02d-cd88f89725c4: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 19_d7236b6d-6335-48eb-bb50-9234138282b2: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 20_2497f6cc-d3d4-4a91-9ce5-10081f4bb70e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 21_00496dcd-e257-44af-b8dd-9c78d5bbc0d6: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 22_6007150f-e57e-49dd-9da3-fdad92bcd31d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 23_113dedaf-f281-403a-b24a-0f0e05cd725c: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 24_a5a2a280-166a-437f-9d02-d57103609435: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 25_b56e1df9-e5fe-4da7-8340-989997a7b742: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 26_5b8c6384-595a-4bd2-9926-4b17cb76b183: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 27_210d5dd6-054f-4971-9ad4-896854b1ff5a: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 28_0a104b07-dda0-4cf8-8b51-6e2b2c09cf04: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 29_f6e0ca39-6816-4767-83be-b52769451466: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 30_c9edf9f5-e54d-439d-8e89-d0ed48afc26c: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 31_2b420999-42a3-4bec-a5b0-0b8ef3f167db: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 32_dacabaa4-1e30-4c1c-9aa7-10573fae3d9e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 33_a23ab402-be93-4cdb-9dfd-7abf9fb63dda: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 34_e718dc11-a88a-403f-87cd-ef81eb7a445c: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 35_370240f6-9bb0-4755-ad35-276f4dba0192: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 36_01379a49-c81b-4b12-a21b-0efa924c54d3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 37_a4a0cfdd-8c6f-4096-a39e-0372bf67cf96: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 38_45bfe0c8-e0dd-46bc-98dc-74e78159a398: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 39_51430ac8-9ff9-448d-92b1-2ce8aba6269b: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 40_ff87d776-a7ab-49f2-beda-fdf33d332da8: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 41_dd6d3d0f-b851-4708-910a-9356595cdcf9: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 42_7a97251c-25aa-48b0-9a1a-3977ae689680: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 43_a1138186-c34a-435b-8913-021ea897abd1: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 44_435cf39e-b62e-4c2c-adc9-454f9e847da2: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 45_fab2e8ea-d4d1-4370-b4b5-5366cbc84f5d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 46_07b1df9f-d52f-49d0-b093-5529ccdd6c55: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 47_1bcada15-5795-4aae-860e-ce24204c57e3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 48_32863e18-62e6-473b-b18f-7820e765d835: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 49_bba9e0da-68d2-4863-ac78-5b13b478bc89: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 50_f27d1928-b45d-4a83-9c7a-45873303f689: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 51_4248492e-c988-48ef-8d58-302d37e3071d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 52_527971bc-dd35-4c1e-ad40-cec19e907251: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 53_a56f7c09-ced8-41d6-a8e3-3ea3570ccc8b: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 54_94549a5f-e2f3-4c8e-801c-d9237b6c865d: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 55_c5f29b56-b7e1-4936-b16c-5e5f7cc478d5: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 56_cb43438c-a87a-4857-9b39-6a6ebb773f51: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 57_0accdada-62b3-4911-9f24-abaf00f3ba62: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 58_64a29108-57e4-4680-b642-dc7531bd2751: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 59_4d41eab2-a9dd-43ca-8d7f-7aeb341e1cdf: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 60_9345d03b-1eb0-4c67-b686-744c303d41b2: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 61_72315ab2-1cf8-4106-8313-5f47cbac7414: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 62_ead5ec60-1c86-420c-b3d4-177e86e745e6: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:31 [omni.py:963] [Orchestrator] Stage 1 error on request 63_1e1a92ff-f120-4573-aef3-e3733ddc189e: EngineCore encountered an issue. See stack trace (above) for the root cause.
(Worker pid=45769) [Stage-1] INFO 02-27 03:29:31 [multiproc_executor.py:732] Parent process exited, terminating worker
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] WorkerProc hit an exception.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Traceback (most recent call last):
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 858, in worker_busy_loop
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] output = func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 361, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return self.worker.execute_model(scheduler_output)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 652, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] output = self.model_runner.execute_model(
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py", line 121, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.synchronize_input_prep(),
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return next(self.gen)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3017, in synchronize_input_prep
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.prepare_inputs_event.synchronize()
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] torch.AcceleratorError: CUDA error: device-side assert triggered
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863]
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Traceback (most recent call last):
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 858, in worker_busy_loop
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] output = func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 361, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return self.worker.execute_model(scheduler_output)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 652, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] output = self.model_runner.execute_model(
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py", line 121, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.synchronize_input_prep(),
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return next(self.gen)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3017, in synchronize_input_prep
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.prepare_inputs_event.synchronize()
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] torch.AcceleratorError: CUDA error: device-side assert triggered
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863]
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863]
(Worker pid=45769) [Stage-1] INFO 02-27 03:29:31 [multiproc_executor.py:785] WorkerProc shutting down.
(EngineCore_DP0 pid=45035) Process EngineCore_DP0:
(EngineCore_DP0 pid=45035) Traceback (most recent call last):
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 652, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] output = self.model_runner.execute_model(
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return func(*args, **kwargs)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py", line 121, in execute_model
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.synchronize_input_prep(),
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] return next(self.gen)
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] ^^^^^^^^^^^^^^
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3017, in synchronize_input_prep
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] self.prepare_inputs_event.synchronize()
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] torch.AcceleratorError: CUDA error: device-side assert triggered
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863]
(Worker pid=45769) [Stage-1] ERROR 02-27 03:29:31 [multiproc_executor.py:863]
(Worker pid=45769) [Stage-1] INFO 02-27 03:29:31 [multiproc_executor.py:785] WorkerProc shutting down.
(EngineCore_DP0 pid=45035) Process EngineCore_DP0:
(EngineCore_DP0 pid=45035) Traceback (most recent call last):
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=45035) self.run()
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=45035) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1010, in run_engine_core
(EngineCore_DP0 pid=45035) raise e
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 999, in run_engine_core
(EngineCore_DP0 pid=45035) engine_core.run_busy_loop()
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1026, in run_busy_loop
(EngineCore_DP0 pid=45035) self._process_engine_step()
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1060, in _process_engine_step
(EngineCore_DP0 pid=45035) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=45035) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 514, in step_with_batch_queue
(EngineCore_DP0 pid=45035) model_output = future.result()
(EngineCore_DP0 pid=45035) ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 81, in result
(EngineCore_DP0 pid=45035) return super().result()
(EngineCore_DP0 pid=45035) ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP0 pid=45035) return self.__get_result()
(EngineCore_DP0 pid=45035) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP0 pid=45035) raise self._exception
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 85, in wait_for_response
(EngineCore_DP0 pid=45035) response = self.aggregate(get_response())
(EngineCore_DP0 pid=45035) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=45035) File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 358, in get_response
(EngineCore_DP0 pid=45035) raise RuntimeError(
(EngineCore_DP0 pid=45035) RuntimeError: Worker failed with error 'CUDA error: device-side assert triggered
(EngineCore_DP0 pid=45035) Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=45035) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=45035) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=45035) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=45035) ', please check the stack trace above for the root cause
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] Failed on batch ['64_2fba6ea3-3570-482d-8d5c-4f7bcd22f586', '65_99e5e1fd-b6e7-435e-9f1e-8f612a518e0e', '66_f59f3c13-c74b-4fe7-a763-cf4d19eda7be', '67_368addad-b230-4ce3-a53f-27278901e977', '68_cadefbc6-ed3e-4cdf-8530-4fb8c89d27a2', '69_8ccde8f0-7ef5-4ad8-92b5-5203fa88c306', '70_d15e052f-814c-4e1b-98f4-7727f99b2a1f', '71_75330d2a-021c-467c-9f93-9388eec1b152', '72_b9935142-7bb9-4c88-8b81-3e2b069cc7dd', '73_a7229367-ee0d-404f-a90e-ce7b1f493302', '74_643a6950-a0ee-4814-b03b-1ec232c4bcff', '75_c4ee1422-55c6-40bb-a302-75087b78cf9e', '76_48f4ee62-66ef-468d-ad22-11eabda5e2e2', '77_ad970c15-6a28-4527-9eea-f573903a3da9', '78_ea1b15c3-901f-4a9e-be26-4b40849b88af', '79_76afb2e1-8d34-4987-b008-f02fabfc1894', '80_6def26a6-ff08-49e3-b066-d31fe498c923', '81_48ad4ff4-8ecd-426c-9378-989d239cb947', '82_0e044f89-02df-418d-9fdf-25d3c809418e', '83_301f4c2b-dabb-4e29-8abe-ce7d772cb610', '84_91acbef3-607c-4aef-8b6e-6c0060aa05d4', '85_071dbffd-a92e-4560-8055-dc6bfd143dd9', '86_f04d1d46-f139-411c-82a6-0cc9a9c1a925', '87_f2fabc1d-82f4-4626-881a-83089c050f6e', '88_6939952a-9187-433f-952f-0ec60f9f9035', '89_f7141791-147c-4972-b51e-d6a895abdd23', '90_b8c0bd43-e7c8-4ab9-b3cf-08b6fb5dfc31', '91_80bf3fcf-40ab-4261-80d8-5bcae1041042', '92_dd91c899-01ea-412c-9e15-fc6381e6c0b3', '93_48be60a3-535e-4c93-90f3-c5b3bfea61ef', '94_5daadcbc-1040-49e4-922c-5647d4bc7c93', '95_37d02235-cc1b-4d32-b321-e6ac19d05078', '96_2cd9dec9-f0e2-424b-848f-5160f42c07c3', '97_01588b49-c488-4fd1-92c0-2a3d8ed9386e', '98_85bb98b7-b88f-4652-a44d-3cb194278a3c', '99_037690db-cbda-4321-9958-0777601eeba3']: EngineCore encountered an issue. See stack trace (above) for the root cause.
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] Traceback (most recent call last):
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/vllm-omni/vllm_omni/entrypoints/omni_stage.py", line 968, in _stage_worker
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] results = stage_engine.generate(
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 449, in generate
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] outputs = self._run_completion(
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1749, in _run_completion
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] self._validate_and_add_requests(
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1848, in _validate_and_add_requests
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] raise e
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1837, in _validate_and_add_requests
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] request_id = self._add_request(
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] ^^^^^^^^^^^^^^^^^^
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1889, in _add_request
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] self.llm_engine.add_request(
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 268, in add_request
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] self.engine_core.add_request(request)
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 761, in add_request
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] self._send_input(EngineCoreRequestType.ADD, request)
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 734, in _send_input
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] self.ensure_alive()
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] File "/home/gcpuser/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 576, in ensure_alive
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] raise EngineDeadError()
[Stage-1] ERROR 02-27 03:29:41 [omni_stage.py:1048] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 64_2fba6ea3-3570-482d-8d5c-4f7bcd22f586: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 65_99e5e1fd-b6e7-435e-9f1e-8f612a518e0e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 66_f59f3c13-c74b-4fe7-a763-cf4d19eda7be: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 67_368addad-b230-4ce3-a53f-27278901e977: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 68_cadefbc6-ed3e-4cdf-8530-4fb8c89d27a2: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 69_8ccde8f0-7ef5-4ad8-92b5-5203fa88c306: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 70_d15e052f-814c-4e1b-98f4-7727f99b2a1f: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 71_75330d2a-021c-467c-9f93-9388eec1b152: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 72_b9935142-7bb9-4c88-8b81-3e2b069cc7dd: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 73_a7229367-ee0d-404f-a90e-ce7b1f493302: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 74_643a6950-a0ee-4814-b03b-1ec232c4bcff: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 75_c4ee1422-55c6-40bb-a302-75087b78cf9e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 76_48f4ee62-66ef-468d-ad22-11eabda5e2e2: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 77_ad970c15-6a28-4527-9eea-f573903a3da9: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 78_ea1b15c3-901f-4a9e-be26-4b40849b88af: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 79_76afb2e1-8d34-4987-b008-f02fabfc1894: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 80_6def26a6-ff08-49e3-b066-d31fe498c923: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 81_48ad4ff4-8ecd-426c-9378-989d239cb947: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 82_0e044f89-02df-418d-9fdf-25d3c809418e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 83_301f4c2b-dabb-4e29-8abe-ce7d772cb610: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 84_91acbef3-607c-4aef-8b6e-6c0060aa05d4: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 85_071dbffd-a92e-4560-8055-dc6bfd143dd9: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 86_f04d1d46-f139-411c-82a6-0cc9a9c1a925: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 87_f2fabc1d-82f4-4626-881a-83089c050f6e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 88_6939952a-9187-433f-952f-0ec60f9f9035: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 89_f7141791-147c-4972-b51e-d6a895abdd23: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 90_b8c0bd43-e7c8-4ab9-b3cf-08b6fb5dfc31: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 91_80bf3fcf-40ab-4261-80d8-5bcae1041042: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 92_dd91c899-01ea-412c-9e15-fc6381e6c0b3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 93_48be60a3-535e-4c93-90f3-c5b3bfea61ef: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 94_5daadcbc-1040-49e4-922c-5647d4bc7c93: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 95_37d02235-cc1b-4d32-b321-e6ac19d05078: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 96_2cd9dec9-f0e2-424b-848f-5160f42c07c3: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 97_01588b49-c488-4fd1-92c0-2a3d8ed9386e: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 98_85bb98b7-b88f-4652-a44d-3cb194278a3c: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 02-27 03:29:41 [omni.py:963] [Orchestrator] Stage 1 error on request 99_037690db-cbda-4321-9958-0777601eeba3: EngineCore encountered an issue. See stack trace (above) for the root cause.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.