ValueError: Requested tokens (590) exceed context window of 512

[dennis-schadeck]

I'm using the local_model.
When I try to execute, the error occurs.
"[..]
llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32
llama_new_context_with_model: n_ctx_per_seq (512) < n_ctx_train (4096) -- the full capacity of the model will not be utilized
[..]
ValueError: Requested tokens (590) exceed context window of 512 "

Is there any way to set the parameters?

[fynnfluegge]

Hi, it is hardcoded in llm.py. I don't actively develop this anymore, but open to merge PRs. If you need this, I am open to merge your PR 🙌

[dennis-schadeck]

Hi, thanks 👍🏼I’ve already submitted a PR for this, so feel free to take a look.