How do we increase the context window? Or do we even need to?

[robdodson]

See this thread for more info:
ollama/ollama#2714

ollama has a num_ctx property which defaults to 4096 tokens. Do we need to increase this?

[g0t4]

Watch for a message like this in your logs when you request completions:

llama_new_context_with_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized

By the way, AFAICT num_ctx defaults to 2048, so 4096 is probably coming from model(s) you are using.

If so then you can adjust num_ctx a few diff ways I've found:

• If using /api/generate see the num_ctx param
• OLLAMA_CONTEXT_LENGTH=8192 ollama serve when starting a server
  
• In a Modelfile using PARAMETER num_ctx <context size> to default to a different value so you don't need the above when using it.
• When using OpenAI compatible endpoints (/v1/completions//v1/chat/completions) then you cannot pass num_ctx, maybe that changes in the future, for now that's one reason why you would entertain the latter two approaches.

Startup logs

To validate current settings for a given model, when a model loads you'll find a section like the following:

llama_init_from_model: n_seq_max     = 4
llama_init_from_model: n_ctx         = 32768
llama_init_from_model: n_ctx_per_seq = 8192
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 1000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_per_seq (8192) < n_ctx_train (32768) -- the full capacity of the model will not be utilized

• n_ctx maps to num_ctx
• n_seq_max is a parallelism parameter => n_ctx_per_seq = n_ctx / n_seq_max, separate dimension that puts a cap on how big n_ctx_per_seq can be which is what dictates how much of a given prompt is truncated in the messsage way up at the top that I referenced.