"Get context window" detection fails; fallback to 4096

[ML-CSI]

When chatting with my transcription I get an error message like this:

"Transcript alone exceeds model context limit (5601 tokens > 4096). Please use a model with larger context window."

disclaimer: I'm new to Scriberr, Github and also writing issues; English is not my main language; doing my best anyway ;)
Please ask if infos are missing :) I'm willing to help as best as possible!

this issue is in a way related to #287 (but not the same!)

I tried all of them with OpenAI as provider in settings

• a) my local OpenWebUI instance's API
• b) OpenRouter
• c) groq
• d) Google AI Studio

...in all the cases I got the same error message above
This indicates that the context window is not detected properly from those platforms.

I checked the code of Scriberr for the error message and found this:
(Scriberr-main\internal\api\chat_handlers.go)

// Get context window
	contextWindow, err := svc.GetContextWindow(c.Request.Context(), session.Model)
	if err != nil {
		fmt.Printf("Failed to get context window for model %s: %v. Using default 4096.\n", session.Model, err)
		contextWindow = 4096
	}

It defaults back to 4096 (maybe because of the old default value of Ollama?)
This explains my problems as later in the code it looks like this (same file as mentioned above)

	// Final check - if still over limit after trimming all possible messages, return error
	if currentTokenCount > contextWindow {
		c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Transcript alone exceeds model context limit (%d tokens > %d). Please use a model with larger context window.", currentTokenCount, contextWindow)})
		return
	}

This equals my error message.

So what could be a solution? Defaulting for a higher context window? 4096 isnt a reasonable setting for running a LLM anyway ;)
Another solutIution? dont know.

[rishikanthc]

So currently, Scriberr detects model context size using the OpenAI API / Ollama API.. specifically this one: https://docs.ollama.com/api-reference/show-model-details
So if your provider doesn't have this endpoint then automatic context size detection would fail.

I'm thinking, I could provide a UI element for the users to manually set the context size for the model with a default fallback.
Also I completely agree that 4096 isn't a reasonable setting by any means for modern LLMs xD

Do you think an option to manually configure context size would be an acceptable solution for this ?

[ML-CSI]

First of all: (Thanks for this great project!)
I feel honored being asked to add my 2 cents...so here they are:

• input field for the number
• Checkbox "override context window"
• hint: "this can cause major problems when set too high for your LLM of choice"
  (all together)

Someone else with other/better ideas?

side notes:
Regarding the fail of detection: I suspected something like this to be the cause. Your idea, detecting context window size, was/is for a good reason.

And to answer your question: I think it would be an improvement as this enables users like me to chat with their transcriptions. Other than that this whole feature is in a way blocked from being used.

As soon as there is a solution in place I would personally set the value to something like 20K-32K to not be in the need for adjusting it too often for every model OR stick with just one LLM and set the value accodingly.
If I understood the code right there are ways implemented to shrink the context window if there is a back and forth conversation. This would help keeping the context window tight.

[rishikanthc]

@ML-CSI thanks for your input. I'll implement and push out a patch 1.2.1
And yes, the code automatically buffers the context.. so if you exceed context window, the older messages are removed from the context automatically..