Help: Exceed 4095 Tokens on Custom Endpoint

[Koesn]

error: [handleAbortError] AI response error; aborting request: Prompt token count of 13620 exceeds max token count of 4095.

I'm using Docker and already setup librechat.yaml to access my home endpoint running miqu 70b 32k ctx. It is works but very limiting max tokens. It also wrong max tokens on Groq mixtral 32k, only max at 8k.

How to make it accepting 32k tokens? Is there any settings to change that limiting 4095 max tokens? Thank's.

Update:
Found my temporary fix. This error throwing in LibreChat/api/app/clients/BaseClient.js and I manually fixed this with:

  async getMessagesWithinTokenLimit(_messages, maxContextTokens) {
    // Every reply is primed with <|start|>assistant<|message|>, so we
    // start with 3 tokens for the label after all messages have been counted.
    let currentTokenCount = 3;
    let summaryIndex = -1;
    // let remainingContextTokens = maxContextTokens ?? this.maxContextTokens; // --> bad "this.maxContextTokens"
    let remainingContextTokens = maxContextTokens ?? 32768; // --> forced my capping, works ootb
    const messages = [..._messages];

I hope there's official fix for this.

[danny-avila]

Not at the moment but I'm actually working on this, this week

[thomasgauthier]

@danny-avila Any news on this front? A way to disable all local context window management and let the remote chat API do it would be great, especially now that we're getting longer and longer contexts and fitting limited windows is not so much of an issue anymore.

[danny-avila]

It’s slated for v0.7.2 check the milestones

[thomasgauthier]

Amazing! Thanks for all the hard work btw. I use LibreChat every day, it's a a fantastic tool.

[danny-avila]

@thomasgauthier thanks I’m glad!

[mostlygeek]

I submitted a PR for together.ai models in #2458. Perhaps something like that would work for groq as well. There doesn't seem to be a naming convention across providers but maintaining a list doesn't seem to be too onerous.

[frenzybiscuit]

Is there a way to set this from the custom model in librechat.yaml?

I can work around this issue by increasing the max context length in the prompt settings as a user, but I'd like the default setting to not be so low. It can be confusing for my end-users.