[Feature request]: Combine models

[MrDrMcCoy]

It would be nice to be able to combine models in the same way we are able to quantize them with stable-diffusion.cpp. This would not only potentially permit folding text encoders, LoRas, etc into a single model file, but also better support certain models that come in split files. Instead of being a separate mode, I would think that it should be an extra option for convert, in the form of --combine-model, which could be specified multiple times.

[SkutteOleg]

I've actually implemented a version of this in my fork yesterday:
https://github.com/SkutteOleg/stable-diffusion.cpp/tree/feature/combine-models-gguf

In convert mode you can specify --diffusion-model, --t5xxl, --vae, etc. and it will produce a single gguf file.

[MrDrMcCoy]

That is great news! However, it would not cover one of the more useful use cases suggested above: Being able to combine split models. If you encounter a text encoder, VAE, or model that comes as a bunch of sequential safetensors files, I think we still need separate functionality to combine those, right?

[lastrosade]

It would be great if we could also do the inverse and separate/remove VAEs/Text Encoders from models.

[leejet]

Merging models isn’t really a great idea when you have several different fine-tuned models. They all share the same text encoders and VAE — only the diffusion part is different. I’m not planning to add this feature to sd.cpp for now, since I’ll be refactoring and optimizing the model loader soon and don’t want to introduce extra complexity. It’s actually something you can do pretty easily with a simple Python script.

[MrDrMcCoy]

I'm reasonably handy with Python. Can you point me to some prior work in this space that would allow for the creation of a simple, low-dependency tool? The main reason I was hoping to find something like this here is that stable-diffusion.cpp doesn't require a massive sprawl of frameworks that are a bear to manage.

[Green-Sky]

Check out "mergekit", I feel like it does everything you need.