Options for supporting multiple users wrt. slots, KV-cache, --prompt-cache, --slot-save

[usrlocalben]

--prompt-cache doesn't seem to do anything, despite having some implementation that reads and populates the value
--slot-save doesn't seem to do anything, and reading the /slot/ handlers I'm not even sure how it's intended to be used.
With -np 1, two users clobber each other's KV-cache, even if users don't mind waiting for a request to be filled. (e.g. 10s of minutes to rebuild a deep ctx slot)
Can KV-cache be (slot-wise) split across CUDA-devs?

Currently it seems like the only way to get two non-clobbering slots would be to avoid GPU use and go CPU only where there's "unlimited" ctx space.

What's the state of affairs wrt. these params and what options are there?

Could an LRU-cache of slots be implemented?

CPU backends don't offer any meaningful concurrency, but swapping out slots would go a long way.

Have I missed something? 😅

[saood06]

What do you mean "--slot-save doesn't seem to do anything". I find it really useful, and am glad I made it work with MLA models as well (see #497), and did and have a UI for managing it with mikupad (still a WIP but screenshots here #558 (comment))

I do agree the state of KV state management has a lot of potential upgrades, I had being able to automatically use any saves in the --slot-save-path as cache (similar to how it caches the prompt now from the RAM). Creating those save files would still be manual (a frontend has the tools to automate it though). Since there is interest I'll move it up the list of things I plan to do.

[usrlocalben]

I've been reading over slot save/restore as I think about a slot disk/mem spill + LRU.

Two things strike me as odd.

1. Due to the asynchrony, there's a race wrt. the state of the slot between time-of-request and time-of-action. One might examine the slot content, then issue save/restore and either get a surprising save-state (slot was modified before save) or accidental invalidation (less harmful, potential delays). save could take the last id_task as a param and give HTTP 409 if upon filling the request the slot's id_task doesn't match.

2. Loading the slot only loads the KV and token list. The prompt text (string) is not restored. This means the current LRU impl. won't match it since it compares by prompt text, so reuse would only occur by the nature of having one slot to choose from. (or issuing restores on all slots)

It seems like either a) rebuilding the prompt text from tokens, or b) storing the prompt string with the slot file would set up the slot functions as being nearly ready to build an LRU spill file.

[saood06]

Thank you for the thoughtful reply.

Loading the slot only loads the KV and token list. The prompt text (string) is not restored. This means the current LRU impl. won't match it since it compares by prompt text, so this system would only work by the nature of having one slot to choose from. (or issuing restores on all slots)

Yes, but from my perspective it is the responsibility of the application to manage the state of the prompt. I don't think what you are describing is a roadblock from this being implemented. You can already do this manually by loading from a slot then sending your prompt, but this would just mean the slot will do the restore for you (there may be extremely fringe situations where it results in performance degradation but rare enough that I think it can still be on by default).

I did recently add a way to peer into what is stored in the so that a more informed decision can be made in my mikupad PR (code here).

In my mikupad PR I was considering automatically pulling the prompt into the currently editing document but ultimately that felt invasive (what if you had a draft in there), I do feel like a copy or insert in button would work but for now copy and paste works and feels fine.

The new UI I pushed yesterday integrates slot and cache management, I would really appreciate if you tried it, because although there are improvements (like discussed above), the new stuff in mikupad works well and does the basics at addressing this problem.

b) storing the prompt string with the slot file would set up the slot functions as being nearly ready to build an LRU spill file.

That already exists? Look at the outputs of the endpoint added in #502 (again this has been updated in my mikupad PR with more stuff see the code for that version here)

On related topics, I was thinking about this as well as ggml-org/llama.cpp#12067.