Token Exceed for librechat but not when directly invoking same model

[amitkewal]

What happened?

Token Exceed for librechat but not when directly invoking model with the same prompt

I have found one issue that I am getting the following error with librechat but when I am directly invoking the same model via streamlit + AWS lambda I am getting a proper response.

{"level":"error","message":"[handleAbortError] AI response error; aborting request: Prompt token count of 37165 exceeds max token count of 4095.","stack":"Error: Prompt token count of 37165 exceeds max token count of 4095.\n at OpenAIClient.handleContextStrategy (/app/api/app/clients/BaseClient.js:374:13)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async OpenAIClient.buildMessages (/app/api/app/clients/OpenAIClient.js:559:61)

Steps to Reproduce

I am working on internal 2 hours transcript data.
So I cant share it here.

What browsers are you seeing the problem on?

No response

Relevant log output

{"level":"error","message":"[handleAbortError] AI response error; aborting request: Prompt token count of 37165 exceeds max token count of 4095.","stack":"Error: Prompt token count of 37165 exceeds max token count of 4095.\n    at OpenAIClient.handleContextStrategy (/app/api/app/clients/BaseClient.js:374:13)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async OpenAIClient.buildMessages (/app/api/app/clients/OpenAIClient.js:559:61)

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Which model, via a custom endpoint? The system has a specific list of models and uses a default context window if it's not recognized.

There will be a way to set the max context via config, but for now you can use the parameters for this:

[AmgadHasan]

@danny-avila Hi
Any updates about this? Can we set the default context window in the configs?

[danny-avila]

Not yet.

[amitkewal]

@danny-avila I am using anthropic.claude-3-5-sonnet-20240620-v1:0

Sharing the librechat yaml.

---

version: 1.0.9
cache: false
interface:
  endpointsMenu: true
  modelSelect: true
  parameters: false
  sidePanel: false
  presets: false
  privacyPolicy:
    externalUrl: https://www.dummy.com/privacy
    openNewTab: true
  termsOfService:
    externalUrl: https://www.dummy.com/terms-of-use
    openNewTab: true

endpoints:
  custom:
    - name: DUMMY
      apiKey: xxxxxxxxxxxxxxxxxxxxxx
      baseURL: http://litellm:8000/v1
      models:
        default:          
          - bedrock/meta.llama3-70b-instruct-v1:0          
        fetch: true
      titleConvo: true
      titleModel: llama3-70b
      summarize: false
      summaryModel: llama3-70b
      forcePrompt: false
      modelDisplayLabel: "AI Chat"
fileConfig:
  endpoints:
    DUMMY: 
      disabled: true

[danny-avila]

That specific model should have context recognized, odd that it isn't for you:

even when I use the exact model identifier:

---

To understand what's going on can you share your debug logs?

If you are using docker, they are saved in the ./logs directory at project root.

Reproduce the issue then check the latest debug logs (files starting with "debug" in name, i.e. logs/debug-2024-09-26.log)

[qymab]

I'm experiencing the issue too.

endpoint: "groq",
  endpointType: "custom",
  resendFiles: true,
    modelOptions.model: "llama-3.3-70b-versatile",
  modelsConfig: "exists",
}
2024 debug: [BaseClient] Loading history:
{
  conversationId: "",
  parentMessageId: "",
}
2024 debug: [BaseClient] Context Count (1/2)
{
  remainingContextTokens: 4092,
  maxContextTokens: 4095,
}
2024 debug: [BaseClient] Difference between original payload (1) and context (0): 1
2024 warn: Prompt token count exceeds max token count (5910 / 4095).
2024 error: [handleAbortError] AI response error; aborting request: { "type": "INPUT_LENGTH", "info": "5910 / 4095" }
2024 debug: [AskController] Request closed

[danny-avila]

warn: Prompt token count exceeds max token coun

This means your input message exceeds the total model context.

I will update it soon for llama-3.3

[djuillard]

Hello,
I face the same max token 4095 issue when I want to use DeepSeek R1 or V3 as a custom endpoint in Azure AI Services (not the serverless service but the AI service for which the endpoint url terminates by /models ans for which we can pass the model parameter the sélect the right one). But when I read the official capabilities of deepseek on Azure AI services, the max context should be higher (https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/models#deepseek). Any tips or workaround ?
Thanks a lot in advance

[danny-avila]

It really depends on how you are naming the model, it must match the system understood name for now

#4259 (comment)

[mlibre]

It would be great to implement the feature to set a maximum context (token limit) per model in the config.yaml file.

[danny-avila]

yes, we are tracking this here: #1633

[mlibre]

Great. Thanks @danny-avila