Consolidation of parameters, and models...

[ihatemakinganaccount]

So, Alpaca is awesome and all, but also, as it's grown, it's becoming a little messy and things are a bit all over the place.

I.
Consider model parameters like top_p and temperature. In Ollama, these are defined in the modelfile, and can also be set with every message generated. While in Alpaca:

• top_p and top_k are settable when making a new model
• temperature is set in instance settings, and works like a forced override of the model's setting (can't be set to nothing)
• context window (num_ctx) is hardcoded at 16384, and even that only for a specific case of basic messages (i.e. not for title generation and other stuff that may generate something)
• max_tokens is an instance setting, but only for instances other than Ollama

So this is a bit disjointed. I'm not sure why you'd want temperature to be an instance setting anyway, since it depends on the model how well it works on a given temperature, and you may want to tweak it per chat even? Or at least leave it to the modelfile. Same thing for context window and max tokens, that depends on what you want from a given model. On one instance, you can run a coding model where you want a huge context window and low creativity, and a fast quirky asistant that just responds to a given command.

What I think would make the most sense:

• have such settings available in the model's creation widget (right now there's top_p and top_k as imagination and focus)
• make them available per-chat or even per-message as selectable overrides. That would need some new UI, but:
  -- there's space in the new chat window, as well as in the '...' menu in the chat bubbles
  -- if these settings were listed there, you could also use that just to see/check their values as 'stats for geeks'
  -- if this would be settable per-message, it would open the door for further improvements down the line, like dynamic temperature settings, which some frontends apparently have
  -- these could be advanced options which you could enable/hide in Preferences if you prefer to have a clean default UI

I'm not sure how other instances than Ollama work, I assume the exact parameters tend to differ, but that's the case with instance settings already (max_tokens seems to be the same as num_predict I guess)

I think this would simplify things in the long run, if you'd want to add more features.

II.
The other thing I'm wondering about, is models, where, at this time:

1. You choose the LLM in the drop-down on top of the window
2. TTS voice and STT size gets chosen in Preferences > Audio, and each gets downloaded only when triggered
3. Then TTS voice can be assigned to a LLM in the LLM's specific settings in the Models window
4. All these are clumped together in the model manager
5. Dictation is, again, a global setting in preferences

I'm not really sure what would be the best approach, I don't know what your plans are. But I think at least for the audio stuff, these shouldn't be hidden under preferences. Especially not the TTS voice, because it's a drag trying out different voices, and you may want to enable dictation only for certain models/chats.

I'd probably make it as new drop-down next to the model selector - so separate model selectors for LLM, TTS and STT.

You can look how Speech Note (https://github.com/mkiol/dsnote) does things, including populating the model database. I think it would make sense to treat the voice models similarly to LLMs. Honestly if it were up to me, I'd try to hook up with Speech Note and use their app for the voice stuff, instead of reinventing everything.

Anyway, just some thoughts I wanted to get out. Thanks for Alpaca!