support ollama num_ctx options

[bb8boy]

Ollama's call uses Openai compatible API, because this API interface has a maximum context of 2K restrictions, resulting in abnormal use. You need to change to the use of Ollama's own API (non -compatible Openai) call. This interface supports the context size settings (num_ctx). You can also use the official Python library
ollama valid-parameters-and-values

[fanghuaqi]

It can be done like this https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size

When using the API, specify the num_ctx parameter:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "options": {
    "num_ctx": 4096
  }
}'

chat_completion = await client.chat.completions.create(
            messages=messages,
            model=model,
            temperature=temperature,
            stream=stream,
            stop=stop_tokens,
            extra_body={"options": {"num_ctx": 64_000}},
            **kwargs,
        )

[mrubens]

Sounds great, we should do this!