[Question]: Setting maxPromptTokens variable - no documentation, but within the code

[ventz]

What is your question?

Hi - curious how/where one would set the variable maxPromptTokens for OpenAI for example?

More Details

This came about because we noticed that gpt4-turbo-preview is set to a 8K input context window, however the model supports 128K.

Not sure if this is a mistake, or intentional, but we want to set it to 128K

Thank you!

What is the main subject of your question?

Endpoints

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Hi thanks for bringing this up.

What is the name of the model when you select it from the web UI? As long as you include gpt-4-turbo anywhere in the name it should pick up on the right context size.

On OpenAI Endpoint

On Azure Endpoint

[ventz]

@danny-avila Thank you for looking into this!

I pulled the latest github version, and noticed that it's correctly showing in: api/utils/tokens.js

'gpt-4-1106': 127990, // -10 from max
'gpt-4-0125': 127990, // -10 from max
'gpt-4-turbo': 127990, // -10 from max

but there is no 'gpt-4-turbo-preview'

I cannot replicate it with the latest pulled version (latest branch from github), however, I do have the debug log from the previous version and the run:

{
  text: "I am an industrial organizational psychologist and I'm currently analyzing employee responses to a q... [truncated]",
  conversationId: null,
  endpoint: "openAI",
  chatGptLabel: null,
  promptPrefix: null,
    modelOptions.model: "gpt-4-turbo-preview",
    modelOptions.temperature: 1,
    modelOptions.top_p: 1,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
}
2024-03-01 20:41:51 debug: [BaseClient] Loading history:
{
  conversationId: "<redacted>-c6ff-4e57-a3d8-18d53547c1e1",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
}
2024-03-01 20:41:51 debug: [BaseClient] Context Count (1/2)
{
  remainingContextTokens: 8188,
  maxContextTokens: 8191,
}
2024-03-01 20:41:51 debug: [BaseClient] Difference between original payload (1) and context (0): 1
2024-03-01 20:41:51 error: [handleAbortError] AI response error; aborting request: Prompt token count of 11072 exceeds max token count of 8191.

You can see the model is gpt-4-turbo-preview and the max token count seems to be max token count of 8191.

Is it possible that the gpt 4 turbo preview is not linking to the "gpt-4-1106" nor "gpt-4-turbo" (which from the tokens.js, both look correct)?

LibreChat's

OpenAI's

[ventz]

Running locally via a container for testing -- docker-compose up on localhost:3080

[danny-avila]

Are you setting the models with the .env file? Can you share how you set them up, it could be some added spaces between commas. I'm going to make it more foolproof so you can add spaces.

[danny-avila]

Running locally via a container for testing -- docker-compose up on localhost:3080

Scratch the models issue, I just tried reproducing that way and it wouldn't work anyway.

Can you try

docker-compose pull
docker-compose build --no-cache
docker-compose up

[ventz]

This is exactly what I had done, and after that it seemed to work. (this is the new env that I cannot reproduce the error in).

[danny-avila]

@ventz glad to hear it's working now!

[CXwudi]

For me I got maxContextToken to 63995 when using gpt-4-turbo-preview, not sure why.

2024-03-04 19:59:02 debug: [AskController]
{
  text: "Hi, how are you",
  conversationId: null,
  endpoint: "openAI",
  chatGptLabel: null,
  promptPrefix: null,
  resendImages: false,
  imageDetail: "auto",
    modelOptions.model: "gpt-4-turbo-preview",
    modelOptions.temperature: 0.5,
    modelOptions.top_p: 1,
    modelOptions.presence_penalty: 0,
    modelOptions.frequency_penalty: 0,
}
2024-03-04 19:59:02 debug: [OpenAIClient] maxContextTokens 63995
2024-03-04 19:59:02 debug: [BaseClient] Loading history:
{
  conversationId: "c101e0ef-9469-4472-8242-b1c0118ab448",
  parentMessageId: "00000000-0000-0000-0000-000000000000",
}
2024-03-04 19:59:02 debug: [BaseClient] Context Count (1/2)
{
  remainingContextTokens: 63983,
  maxContextTokens: 63995,
}
2024-03-04 19:59:02 debug: [BaseClient] Context Count (2/2)
{
  remainingContextTokens: 63983,
  maxContextTokens: 63995,
}

My OpenAI setting is:

OPENAI_API_KEY=<your key>
OPENAI_MODELS=gpt-4-vision-preview,gpt-4-turbo-preview,gpt-3.5-turbo-0613,gpt-3.5-turbo-instruct-0914,gpt-3.5-turbo-instruct,gpt-3.5-turbo-0125,gpt-3.5-turbo,gpt-3.5-turbo-0301,gpt-3.5-turbo-1106,gpt-3.5-turbo-16k,gpt-3.5-turbo-16k-0613,gpt-4-1106-preview,gpt-4-0125-preview
 # I simply just removed the non turbo gpt-4 from the model list

DEBUG_OPENAI=true

TITLE_CONVO=true
OPENAI_TITLE_MODEL=gpt-3.5-turbo

OPENAI_SUMMARIZE=true
OPENAI_SUMMARY_MODEL=gpt-3.5-turbo

And I am using the latest ghcr.io/danny-avila/librechat-dev image upon writing

[danny-avila]

this makes sense actually, I made it to half of the model's context to accommodate for models with very large context windows.

[6b6a72]

On a related note, what is the recommended location for setting the max token value for a custom model using the chat UI?

It looks like it's all based on the name or a default - do I just need to do it in the script? Thanks!

[danny-avila]

what is the recommended location for setting the max token value for a custom model using the chat UI

There is none yet. This is planned

[CXwudi]

Related #2000