how to setup config.yaml as vllm service(using local model) proxy

[liyuerich]

1 、I have 2 vllm service on different container， using local model , which located at /root/data/Qwen/Qwen2-0.5B-Instruct

2、 run vllm serve /root/data/Qwen/Qwen2-0.5B-Instruct successfully, could get vllm model list

3、set up litellm config.yaml with one vllm service

model_list:

• model_name: "vllm-model-group-1"
  litellm_params:
  model: "vllm/root/data/Qwen/Qwen2-0.5B-Instruct"
  api_base: "http://xxx.xxx.xxx.xxx:8000/v1"
  request_timeout: 120

4、 litellm --config config.yaml --detailed_debug

the debug info is as followed:

root@ins-98fj6-656c84fb97-n9qms:~/data# litellm --config litellm_config.yaml --detailed_debug
INFO: Started server process [13153]
INFO: Waiting for application startup.
21:30:20 - LiteLLM Proxy:DEBUG: proxy_server.py:557 - litellm.proxy.proxy_server.py::startup() - CHECKING PREMIUM USER - False
21:30:20 - LiteLLM Proxy:DEBUG: litellm_license.py:102 - litellm.proxy.auth.litellm_license.py::is_premium() - ENTERING 'IS_PREMIUM' - LiteLLM License=None
21:30:20 - LiteLLM Proxy:DEBUG: litellm_license.py:111 - litellm.proxy.auth.litellm_license.py::is_premium() - Updated 'self.license_str' - None
21:30:20 - LiteLLM Proxy:DEBUG: proxy_server.py:570 - worker_config: {"model": null, "alias": null, "api_base": null, "api_version": "2024-07-01-preview", "debug": false, "detailed_debug": true, "temperature": null, "max_tokens": null, "request_timeout": null, "max_budget": null, "telemetry": true, "drop_params": false, "add_function_to_prompt": false, "headers": null, "save": false, "config": "litellm_config.yaml", "use_queue": false}

LiteLLM: Proxy initialized with Config, Set models:
vllm-model-group-1
21:30:20 - LiteLLM:DEBUG: utils.py:4710 - checking potential_model_names in litellm.model_cost: {'split_model': '17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a', 'combined_model_name': '17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a', 'stripped_model_name': '17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a', 'combined_stripped_model_name': '17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a', 'custom_llm_provider': None}
21:30:20 - LiteLLM:DEBUG: utils.py:4925 - Error getting model info: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
21:30:20 - LiteLLM:DEBUG: utils.py:2335 - added/updated model=17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a in litellm.model_cost: 17f4931880bfed84105b516a0136b55391c52aeb8be20b0271a381c01c6d3f5a
21:30:20 - LiteLLM:DEBUG: utils.py:4710 - checking potential_model_names in litellm.model_cost: {'split_model': 'root/data/Qwen/Qwen2-0.5B-Instruct', 'combined_model_name': 'vllm/root/data/Qwen/Qwen2-0.5B-Instruct', 'stripped_model_name': 'vllm/root/data/Qwen/Qwen2-0.5B-Instruct', 'combined_stripped_model_name': 'vllm/root/data/Qwen/Qwen2-0.5B-Instruct', 'custom_llm_provider': 'vllm'}
21:30:20 - LiteLLM:DEBUG: utils.py:4925 - Error getting model info: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
21:30:20 - LiteLLM:DEBUG: utils.py:2335 - added/updated model=vllm/root/data/Qwen/Qwen2-0.5B-Instruct in litellm.model_cost: vllm/root/data/Qwen/Qwen2-0.5B-Instruct
21:30:20 - LiteLLM Router:DEBUG: router.py:5098 -
Initialized Model List ['vllm-model-group-1']
21:30:20 - LiteLLM Router:INFO: router.py:691 - Routing strategy: simple-shuffle
21:30:20 - LiteLLM Router:DEBUG: router.py:559 - Intialized router with Routing strategy: simple-shuffle

Routing enable_pre_call_checks: False

Routing fallbacks: None

Routing content fallbacks: None

Routing context window fallbacks: None

Router Redis Caching=None

21:30:20 - LiteLLM Proxy:DEBUG: proxy_server.py:644 - prisma_client: None
21:30:20 - LiteLLM Proxy:DEBUG: hanging_request_check.py:148 - Checking for hanging requests....
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:41691 (Press CTRL+C to quit)

[liyuerich]

run command line: litellm-proxy models list

there is no response

[liyuerich]

anything else I could do?

[liyuerich]

it is resolved. thanks.