Does paper-qa support vllm 0.7.3?

[linkai1208]

I have set up vllm on Ubuntu 22.04, and I can already access the vllm model via requests from another computer. However, I want PaperQA to use the vllm model. How do I create a Settings object for this purpose?

[dosubot[bot]]

Hi @linkai1208! I'm Dosu and I’m helping the paper-qa team.

Yes, PaperQA can use a vllm 0.7.3 server as long as it exposes an OpenAI-compatible API endpoint. PaperQA delegates LLM connections to LiteLLM, which supports vllm and other OpenAI-compatible providers. You just need to configure the Settings object with your vllm endpoint and model name.

Here’s an example of how to set this up:

from paperqa import Settings

settings = Settings(
    llm="your-vllm-model",
    llm_config={
        "model_list": [
            {
                "model_name": "your-vllm-model",
                "litellm_params": {
                    "model": "your-vllm-model",
                    "api_base": "http://your-vllm-server:8000/v1",
                    "api_key": "your-api-key-if-needed",
                    "temperature": 0.1,
                    "max_tokens": 512
                }
            }
        ]
    },
    summary_llm="your-vllm-model",
    summary_llm_config={...},  # same as llm_config if you want
    embedding="your-embedding-model",
    embedding_config={...}  # specify embedding model and endpoint if not using OpenAI
)

Make sure to set the embedding model explicitly, since PaperQA defaults to OpenAI embeddings if not specified. You can find more details and examples in the README and in issue discussions.

There are no recent changes or caveats in PaperQA affecting vllm integration—compatibility is determined by LiteLLM’s support for your vllm API version.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[linkai1208]

Thank you for your response, but after following your suggestions, I received the following error message:

Encountered exception during tool call for tool gen_answer: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=deepseek-llama3-70b
Pass model as E.g. For 'Huggingface' inference endpoints pass in completion(model='huggingface/starcoder',..)

How should I configure this LLM provider? Your answer would be greatly appreciated. @dosu

[dosubot[bot]]

The error happens because LiteLLM requires the provider to be included as a prefix in the model string. Since your vllm server exposes an OpenAI-compatible API, you should use the prefix openai/ in your model configuration.

Try this in your PaperQA Settings:

from paperqa import Settings

settings = Settings(
    llm="openai/deepseek-llama3-70b",
    llm_config={
        "model_list": [
            {
                "model_name": "openai/deepseek-llama3-70b",
                "litellm_params": {
                    "model": "openai/deepseek-llama3-70b",
                    "api_base": "http://your-vllm-server:8000/v1",
                    "api_key": "your-api-key-if-needed",
                    "temperature": 0.1,
                    "max_tokens": 512
                }
            }
        ]
    },
    # Set summary_llm and embedding as needed
)

The key is to use "openai/deepseek-llama3-70b" (not just "deepseek-llama3-70b") as the model name and in litellm_params["model"]. This tells LiteLLM to treat your endpoint as an OpenAI-compatible provider and routes requests correctly, resolving the "LLM Provider NOT provided" error reference reference.

If you use a different provider, change the prefix accordingly (e.g., "huggingface/...", "deepseek/...", etc.), but for vllm with OpenAI API, "openai/" is correct. Make sure to also set your embedding model explicitly if you don't want to use OpenAI embeddings reference.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[linkai1208]

Adding a custom_llm_provider is enough, thank you for your reminder!

settings = Settings(
llm="deepseek-llama3-70b",
llm_config={
"model_list": [
{
"model_name": "deepseek-llama3-70b",
"litellm_params": {
"model": "deepseek-llama3-70b",
"api_base": "http://your-vllm-server:8000/v1",
"api_key": "your-api-key-if-needed",
"custom_llm_provider": "openai",
"temperature": 0.1,
"max_tokens": 512
}
}
]
},
# Set summary_llm and embedding as needed
)