Issue with Databricks Claude 3.7 Serving Endpoint in librechat.yaml

[Huguetteloulou]

Hi,

I’ve encountered an issue when attempting to use the Claude 3.7 serving endpoint from Databricks with LibreChat. While I was able to successfully set up and chat with the databricks-meta-llama-3-3-70b-instruct model using the librechat.yaml configuration file, the Claude 3.7 endpoint does not work properly.

Here’s what I’ve done so far:

1. I added the Claude 3.7 endpoint to the custom section of the librechat.yaml file, following the same structure I used for the databricks-meta-llama-3-3-70b-instruct model.
2. The Claude 3.7 model appears in the LibreChat interface as expected.
3. However, when I try to chat with the Claude 3.7 model, it doesn’t respond or function, despite the fact that the endpoint works fine when hosted directly in Databricks.

For reference, here’s the relevant portion of my librechat.yaml configuration:

custom:
  - name: 'Databricks-Llama'
    apiKey: 'XXXXX'
    baseURL: 'XXXXX>/serving-endpoints/databricks-meta-llama-3-3-70b-instruct/invocations'
    models:
      default: [
          "databricks-meta-llama-3-3-70b-instruct",
        ]
      fetch: false
    titleConvo: true
    titleModel: 'databricks/meta-llama-3-3-70b-instruct'
    modelDisplayLabel: 'Meta Llama 3.3 70B Instruct'
    directEndpoint: true # required
    titleMessageRole: 'user' # required
  - name: 'Databricks-Claude'
    apiKey: 'XXXXX'
    baseURL: 'XXXXX/serving-endpoints/databricks-claude-3-7-sonnet/invocations'
    models:
      default: [
          "databricks-claude-3-7-sonnet",
        ]
      fetch: false
    titleConvo: true
    titleModel: "databricks/claude-3-7-sonnet"
    modelDisplayLabel: 'Claude 3.7 Sonnet' 
    directEndpoint: true # required
    titleMessageRole: 'user' # required

As you can see, the configuration for Claude 3.7 is almost identical to that of the Meta Llama 3.3 model, except for the endpoint and model details.

Request:
Could you help me identify what might be going wrong here? Are there additional settings or requirements specific to the Databricks Claude 3.7 endpoint that I might be missing? Any guidance would be greatly appreciated! Let me know if more details or logs are needed!

[ion-elgreco]

Same for me, it seems that Claude 3.7 requires different configuration in a Databricks environment vs the other models hosted in databricks 🤷‍♂️