Not able to connect to Databricks

[aiUser1001]

I've followed all the instructions to add a custom LLM endpoint. I was able to make it the default one but I am unsuccessful in making it work. I am trying to connect to the Databricks serving endpoints, namely databricks-meta-llama-3-70b-instruct. I got the URL as well the api token. It works in curl but it doesn't in LibreChat. Here is the error:

error: [handleAbortError] AI response error; aborting request: 404 status code (no body)

I've checked the debug log and when I compare openAI (which works) to the Databricks one, they look similar when it comes to the data structure. Here is the output from the debug log:

2024-05-23T21:40:43.677Z debug: [OpenAIClient] chatCompletion
{
baseURL: "https://REDACTED.azuredatabricks.net/serving-endpoints/databricks-meta-llama-3-70b-instruct/in... [truncated]",
modelOptions.model: "databricks-meta-llama-3-70b-instruct",
modelOptions.temperature: 1,
modelOptions.top_p: 1,
modelOptions.presence_penalty: 0,
modelOptions.frequency_penalty: 0,
modelOptions.stop: undefined,
modelOptions.max_tokens: 500,
modelOptions.user: "REDACTED",
modelOptions.stream: true,
// 1 message(s)
modelOptions.messages: [{"role":"user","content":"hi"}],
}
2024-05-23T21:40:44.044Z warn: [OpenAIClient.chatCompletion][stream] API error
2024-05-23T21:40:44.045Z error: [handleAbortError] AI response error; aborting request: 404 status code (no body)

I've also checked that Databricks follows the OpenAI standard. Your help is much appreciated. Thank you.

[danny-avila]

with Mistral API, this happens when unsupported parameters are sent, so this may be the case with Databricks as well.

try dropping parameters and see if it helps: https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/custom_endpoint#dropparams

[aiUser1001]

Thanks Danny for the quick response. I am looking at the Post request in Postman and it doesn't look like:
"messages": [{"role": "user", "content": "Say this is a test!"}]. It has a lot more many parameters and no "messages" array. Why is the http post payload different or am I missing something here?

[danny-avila]

it's not different, we send other parameters that are supported by OpenAI by default:

https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/default_params

[aiUser1001]

Hi Danny How can I log the Post request before it gets sent? Many thanks

…

Sent from my iPhone On May 24, 2024, at 9:53 AM, Danny Avila ***@***.***> wrote:  it's not different, we send other parameters that are supported by OpenAI by default: https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/default_params — Reply to this email directly, view it on GitHub<#2847 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BIWFVLDRREVLTYPEY7DGO63ZD5A45AVCNFSM6AAAAABIGOWNMOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNBXHE2DS>. You are receiving this because you authored the thread.Message ID: ***@***.***> Alert: This email is from outside of Abacus Insights. Please examine closely before clicking links or opening attachments. When in doubt, report in the #phishing slack channel IMPORTANT NOTICE: This communication, including any attachment, contains information that may be confidential or privileged, and is intended solely for the entity or individual to whom it is addressed. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message is strictly prohibited. Nothing in this email, including any attachment, is intended to be a legally binding signature.

[danny-avila]

if they are not openai-spec compatible, there's no configuration that would work. we would need to develop this for them.

[aiUser1001]

they are open-spec compatible and another user here was able to make it work: #1940.
I am using Fiddler to inspect the headers and the payload and for some reason the Path section serving-endpoints/databricks-meta-llama-3-70b-instruct/invocations after https://adb-REDACTED.azuredatabricks.net/ is missing

[aiUser1001]

and BTW, it works with curl in the Terminal

[danny-avila]

another user here was able to make it work: #1940.

If you are willing to share the credentials for testing, I can try to figure out why it's not working for you.

Email at [email protected] or reach me on discord: http://discord.librechat.ai/

[aiUser1001]

Thanks Danny, I've sent you an email

[danny-avila]

@aiUser1001 getting to the bottom of this. Here's the root issue, your serving endpoint is not technically compatible. OpenAI Spec expects the baseURL to have a /chat/completions path at the end of the route. so it's making this request:

console.log('About to make a request', url, init);

log output:

About to make a request https://${DBRX_ENDPOINT}/invocations/chat/completions {

However, since all it takes for this to be compatible, is to drop /chat/completions, I'm thinking of adding an option for this.

[aiUser1001]

I am not using DBRX, so how did that come out in the log?
also in #1940, the url format is the same as mine and he got it to work. Is there a work around that I can use right away? Thanks

[aiUser1001]

Danny, actually what you said makes sense.

[danny-avila]

I am not using DBRX, so how did that come out in the log?

it was just my way of redacting the endpoint

also in #1940, the url format is the same as mine and he got it to work. Is there a work around that I can use right away? Thanks

it's actually doubtful they got it to work unless they edited the code, or served it on another API service that makes openai-spec compatible

Danny, actually what you said makes sense.

Great, I'm testing ways of adding this as an option

[danny-avila]

@aiUser1001

After merging this #2895, the following config values will work for you:

endpoints:
  custom:
    - name: 'Llama3'
      apiKey: '${DATABRICKS_API_KEY}'
      baseURL: 'your_databricks_url_ending_with_invocations'
      models:
        default: [
          "databricks-meta-llama-3-70b-instruct",
        ]
        fetch: false
      titleConvo: true
      titleModel: 'databricks-meta-llama-3-70b-instruct'
      modelDisplayLabel: 'YourModelDisplayLabel'
      titleMessageRole: 'user' # <-----------NEW
      directEndpoint: true # <---------------NEW

• directEndpoint: beneficial for when a custom endpoint's baseURL is the complete endpoint, without needing the appended path of /chat/completions which is the default behavior. This adds compatiblity with API services such as serving-endpoints from Databricks
• titleMessageRole: By default, the instructions payload for titles will use the system role. Some APIs will not allow a system message to be first or alone in a messages payload, and this lets you customize it (as is necessary for databricks)

Will need to update docs after merging.

[danny-avila]

[jainrahulsethi]

Hi Danny, The file upload functionality (chat with file) doesnt seem to be working with databricks? Any particular thing that I am missing?