Whats the context for choosing 30min ttl for serving models?

[sbe-arg]

Is there not a better way to keep loading the model in small chunks of 5min blocks instead of loading it for 30min at the time?

[Patrick-Erichsen]

This is for Ollama? I'm not familiar with the setting

[Patrick-Erichsen]

Is there not a better way to keep loading the model in small chunks of 5min blocks instead of loading it for 30min at the time?

Not that I'm aware of, but open to ideas 😁

[ShaunaGordon]

Yes, it's Ollama. Continue also likes to default to 32768 context size on Ollama, too, even when the model's context isn't that large.