Model Mismatch for Custom Endpoint

[kevindaffaarr]

So I want to use Portkey due to the easiness for monitoring. But I got something strange:

1. LibreChat try to call OpenAI for defining the title. Below is my librechat.yaml snippet, and the screenshots of logs and the web app

# Definition of custom endpoints
endpoints:
  custom:
    # Portkey Google AI Studio
    - name: 'Gemini'
      apiKey: 'dummy'
      baseURL: 'https://api.portkey.ai/v1'
      headers:
        x-portkey-api-key: '${PORTKEY_API_KEY}'
        x-portkey-virtual-key: 'google-virtual-123abc'
      models:
        default: ['gemini-2.0-flash-thinking-exp','gemini-2.0-flash-exp','gemini-1.5-flash','gemini-1.5-pro','gemini-1.5-flash-8b']
        fetch: false
      titleConvo: true
      titleModel: 'gemini-1.5-flash-8b'
      modelDisplayLabel: 'Portkey: Gemini'

LibreChat Screenshot

Logs

The Portkey Virtual Key

2. Then I inspect the Portkey, the model called is correct, but I think this token number is very big that doesn't sounds like the testing chat.

I'm running LibreChat via docker with tutorial for local installation, although I ran it on my Debian server. The changes is on librechat.yaml, and .env for API Keys, and list of endpoint.

Please help me find the problem here.

[danny-avila]

I don't see where you're seeing that OpenAI is being called? Can you share the debug logs found in ./logs at project root?

[kevindaffaarr]

My bad, forgot to paste the most important part. Here is the screenshots and the detail. Appreciate your response!

2025-02-01T03:02:15.594Z debug: [AskController]                                                                           {                                                                                                                           text: "Hello there",
  conversationId: null,
  endpoint: "Gemini",
  endpointType: "custom",                                                                                                   resendFiles: true,
    modelOptions.model: "gemini-2.0-flash-thinking-exp",
  artifactsPrompt: "The assistant can create and reference artifacts during conversations.\n\nArtifacts are for substanti>  modelsConfig: "exists",
}
2025-02-01T03:02:15.596Z debug: [BaseClient] Loading history:
{                                                                                                                           conversationId: "b6c02a89-070d-4a86-96f8-fbe0f071b54d",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
}
2025-02-01T03:02:15.604Z debug: [BaseClient] instructions tokenCount: 2452
2025-02-01T03:02:15.605Z debug: [BaseClient] Context Count (1/2)
{
  remainingContextTokens: 28259,
  maxContextTokens: 30720,
}
2025-02-01T03:02:15.605Z debug: [BaseClient] Context Count (2/2)
{
  remainingContextTokens: 28259,
  maxContextTokens: 30720,
}
2025-02-01T03:02:15.605Z debug: [BaseClient] tokenCountMap:
{
  07a3ca40-e687-43bb-8e66-f3e7a8f9352e: 6,
}
2025-02-01T03:02:15.605Z debug: [BaseClient]
{
  promptTokens: 2461,
  remainingContextTokens: 28259,
  payloadSize: 2,
  maxContextTokens: 30720,
}
2025-02-01T03:02:15.605Z debug: [BaseClient] tokenCountMap
{
  07a3ca40-e687-43bb-8e66-f3e7a8f9352e: 6,
  instructions: 2452,
}
2025-02-01T03:02:15.605Z debug: [BaseClient] userMessage
{
  messageId: "07a3ca40-e687-43bb-8e66-f3e7a8f9352e",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
  conversationId: "b6c02a89-070d-4a86-96f8-fbe0f071b54d",
  sender: "User",
  text: "Hello there",
  isCreatedByUser: true,
  tokenCount: 6,
}
2025-02-01T03:02:15.606Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "https://api.portkey.ai/v1",
    modelOptions.model: "gemini-2.0-flash-thinking-exp",
    modelOptions.user: "679d76e87d24ffe679870928",
    modelOptions.stream: true,
    // 2 message(s)
    modelOptions.messages: [{"role":"system","content":"Instructions:\nThe assistant can create and reference artifacts during c... [truncated],{"role":"user>}
2025-02-01T03:02:15.609Z debug: Making request to https://api.portkey.ai/v1/chat/completions
2025-02-01T03:02:15.615Z debug: [saveConvo] api/app/clients/BaseClient.js - saveMessageToDatabase #saveConvo
2025-02-01T03:02:18.182Z debug: [OpenAIClient] chatCompletion response
{
  object: "chat.completion",
  provider: "google",
    usage.prompt_tokens: 2635,
    usage.completion_tokens: 9,
    usage.total_tokens: 2644,
  id: "google-1738378938085",
  // 1 choice(s)
  choices: [{"message":{"role":"assistant","content":"Hello! How can I help you today?","refusal":null,"parsed":... [truncated]],
  created: 1738378938,
  model: "",
}
2025-02-01T03:02:18.185Z debug: [spendTokens] conversationId: b6c02a89-070d-4a86-96f8-fbe0f071b54d | Context: message | Token usage:
{
  promptTokens: 2461,
  completionTokens: 9,
}
2025-02-01T03:02:18.189Z debug: [spendTokens] No transactions incurred against balance
2025-02-01T03:02:18.200Z debug: [saveConvo] api/app/clients/BaseClient.js - saveMessageToDatabase #saveConvo
2025-02-01T03:02:18.213Z debug: [AskController] Request closed
2025-02-01T03:02:18.217Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "https://api.portkey.ai/v1",
    modelOptions.model: "gemini-1.5-flash-8b",
    modelOptions.user: "679d76e87d24ffe679870928",
    modelOptions.temperature: 0.2,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
    modelOptions.max_tokens: 16,
    // 1 message(s)
    modelOptions.messages: [{"role":"system","content":"Please generate a concise, 5-word-or-less title for the conversation, us... [truncated]],
}
2025-02-01T03:02:18.218Z debug: Making request to https://api.portkey.ai/v1/chat/completions
2025-02-01T03:02:18.502Z warn: [OpenAIClient.chatCompletion][create] API error
2025-02-01T03:02:18.503Z error
2025-02-01T03:02:18.504Z error: [OpenAIClient.chatCompletion] Unhandled error type Error: 400 google error: * GenerateContentRequest.contents: contents is no>2025-02-01T03:02:18.504Z error: [OpenAIClient] There was an issue generating the title with the completion method Error: 400 google error: * GenerateContentR>2025-02-01T03:02:18.505Z debug: [OpenAIClient] Convo Title: New Chat
2025-02-01T03:02:18.505Z debug: [saveConvo] api/server/services/Endpoints/openAI/addTitle.js

[danny-avila]

Thanks for sharing, we can surmise a lot from this.

2025-02-01T03:02:18.218Z debug: Making request to https://api.portkey.ai/v1/chat/completions

This shows that the request is correctly going to your portkey endpoint for each request being made including the title.

The "OpenAI" verbiage is just for the relevant backend part being used, since custom endpoints share functionality with OpenAI due to being "OpenAI-like" or OpenAI API-compatible.

The high tokens come from the default artifacts prompt, which is indeed lengthy and you likely have the feature enabled.

Lastly, the title may be failing as Google may not like a single System message being sent as part of the chat history here:

modelOptions.messages: [{"role":"system","content":"Please generate a concise, 5-word-or-less title for the conversation, us... [truncated]],

OpenAI allows this, but I could think about making a fallback attempt for custom endpoints with a single user role to help resolve this.

[kevindaffaarr]

I see, sorry for assuming the OpenAI part. Thanks for your clear explanation.

[kevindaffaarr]

Duplicate of #4927