OpenLLM deployed in cloud that requires use of Bearer token

[ColinLeverger]

Hello there,

I have deployed an OpenLLM on a managed service that is protected by the use of an auth Bearer token:

curl -X 'POST' \
  'https://themodel.url/v1/generate' \
  -H "Authorization: Bearer Sh6Kh4[ . . . super long bearer token ]W18UiWuzsz+0r+U"
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What is the difference between a pigeon",
  "llm_config": {
    "use[ . . . and so on]
    }
  }'

Curl works like a charm.

In LangChain, I try to create my new llm as such:

llm = OpenLLM(server_url="https://themodel.url/v1/generate", temperature=0.2)

But I don't know how to include this Bearer token, I even suspect it is impossible...

Any clues? :D

Thanks!
Cheers,
Colin

[dosubot[bot]]

🤖

Hello Colin,

You're correct in your suspicion. Currently, the OpenLLM class in the LangChain codebase does not support the use of Bearer tokens for authentication. The class initializes either an HTTPClient or a GrpcClient from the openllm library, depending on the server type, but there is no provision for passing authentication headers in the client initialization or in the query method of the client.

Here's the relevant code for reference:

client_cls = (
    openllm.client.HTTPClient
    if server_type == "http"
    else openllm.client.GrpcClient
)
client = client_cls(server_url)

This is where the client is initialized, and there is no parameter for authentication headers.

if self._client:
    res = self._client.query(prompt, **config.model_dump(flatten=True))

This is where the client's query method is called, and again, there is no parameter for authentication headers.

This is a limitation that we should address. I recommend creating a new issue in the repository to request this feature. In the meantime, you might need to modify the OpenLLM class in your local copy of the LangChain codebase to include the Bearer token in the HTTPClient or GrpcClient initialization and in the query method.

I hope this helps, and I'm sorry for the inconvenience. Let me know if you have any other questions.

Best,
Dosu

Sources

• libs/langchain/langchain/llms/openllm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.