Fix the "AttributeError: 'AsyncLLM' object has no attribute 'engine'" from #3704

surak · web-flow · commit c4452e5ccb1d · 2025-04-08T19:21:04.000+02:00
diff --git a/fastchat/serve/vllm_worker.py b/fastchat/serve/vllm_worker.py
@@ -24,6 +24,9 @@
 )
 from fastchat.utils import get_context_length, is_partial_stop
 
+# This makes vllm > 0.8.0 work again.
+import os
+os.environ['VLLM_USE_V1'] = os.environ.get('VLLM_USE_V1', '0')
 
 app = FastAPI()
 

Original file line number	Diff line number	Diff line change
`@@ -24,6 +24,9 @@`
`24`	`24`	`)`
`25`	`25`	`from fastchat.utils import get_context_length, is_partial_stop`
`26`	`26`
	`27`	`+# This makes vllm > 0.8.0 work again.`
	`28`	`+import os`
	`29`	`+os.environ['VLLM_USE_V1'] = os.environ.get('VLLM_USE_V1', '0')`
`27`	`30`
`28`	`31`	`app = FastAPI()`
`29`	`32`