How to use self-hosted models by API in a way that lets me swap models?

[jerkstorecaller]

Could use some advice here.

Here's my setup:

1. Librechat running on a cloud VPS at librechat.mydomain, that I use with paid APIs like OpenRouter. Let's call this Librechat Server.
2. Linux server idle on my LAN, that I'd like to use to run local GGUF models. It can fit only a single model in RAM. Let's call this GGUF Server.
3. I can make both see each other via a tunnel, no issues there

What can I do to achieve this:

1. be able to place new gguf models on GGUF Server in a directory like ~/gguf_models
2. Librechat somehow lists all the models in GGUF Server, eg Selfhosted -> Model1, Selfhosted -> Model2. (I don't mind manually editing Librechat.yaml for this, I won't add new models often)
3. If I select Selfhosted/Model2 in Librechat, then:
   a. If Model2 is what's already loaded on GGUF Server, returns a quick response
   b. If a different model was loaded, then the previous model is unloaded, and the GGUF Server loads Model2 before answering Librechat (since only one model fits in RAM). Next request to this model goes through step A so it's fast.

So I need knowledge of some other tool than Librechat that's able to do model hotswap based on what Librechat selects.

Is this doable with existing tools?