System Prompt Not Carried to API

[Koesn]

I set custom instructions to send input to Oobabooga API. Seems in Ooba log there's no system prompt carried. Please help.

[danny-avila]

does Oobabooga API support system messages? It would be helpful to share/inspect your logs.

If using docker, the logs are in the ./logs directory from project root. Share only the "debug" logs relevant when reproducing the issue if possible.

[Koesn]

Sure. I tried ooba with --verbose and checked all of the logs. On daily basis, I use client app like MindMac/Chatbox to connect to ooba and also public endpoint like OpenAI/Anyscale, ooba running well. I use large system prompts like 50k-500k tokens on daily basis, so there seems nothing wrong with my dedicated ooba server. Seems LibreChat is not carrying system prompt, not only to Ooba but also to other public endpoint. Or maybe I use a wrong version of Libre? I love the idea of presets and it's web-based.

[danny-avila]

it does correctly send system messages, but some logs would be helpful to understand what's going on

[Koesn]

Okay. I've explored and found that I should setup Max Context Tokens explicitly higher than input's tokens size. Maybe this should be well documented as this is not natural behaviour. Thank's anyway for your attention.

[danny-avila]

LibreChat is not limited in this way, what is likely happening is that the model context is not large enough to include the "system prompt"

What models are you using (the exact value in the frontend)?

To be sure it works as expected, fill this value as needed with the preset you are configuring

[Koesn]

I am using llama-3-gradient with 262k length, and has 47k system prompt. I fill 50k on Max Context Tokens box and it works. I don't know the "System" default is just not working. It also doesn't work on Anyscale and Groq, but Openai and Cohere does.

[danny-avila]

Right that’s because most people will use llama3 with 8k context, the system uses hardcoded values for this