[Question]: Mistral token count

[euggen0106]

What is your question?

Mistral API burns about 100 input tokens just for saying "hello". Am I doing something wrong? Is there any solution to correct this behaviour?

More Details

I did nothing more than creating the endpoint and starting a conversation.

What is the main subject of your question?

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

For the first response in a conversation: yes. It uses about that much to title the conversation.

You can disable it by setting titleConvo: false in librechat.yaml

[euggen0106]

I did set titleConvo: false. From what I can tell, for my every prompt, it also processes all the previous history of the conversation. I asked the API a simple 7 words question and my token count on Mistral platform immediately acounted about 600 input tokens which is all the conversation history combined. Is this the normal behavior?

[danny-avila]

it also processes all the previous history of the conversation ... immediately acounted about 600 input tokens which is all the conversation history combined. Is this the normal behavior?

Yeah, good question. This is normal behavior for all large language models, local or remote (via API). They are not stateless, and need to be provided context to "hold a conversation." It works like this under the hood for every generative text AI platform.

[danny-avila]

The longer a conversation gets, the more tokens it will use; in other words, the larger its context gets. This can be mitigated by the summary feature, which caps your total conversation tokens around 2000 tokens. Otherwise, tokens can reach near the max (32k for Mistral) but tokens will be cut off before exceeding the max so as to avoid API errors.