Local LLM and Local Embedding possible?

[EnviralDesign]

Love this project and it's elegant strategy for becoming a ready to run API endpoint for easy integration into other pipelines!
That alongside the node based UX makes it super ideal for prototyping but also possibly for deployment.

One constraint to using Flowise at the moment is I cannot plug in local embedding models that run inference locally. The project I'm working on is one that involves over a thousand pages of docs, sometimes more depending on the chunking strategy I'm trying to running this through embedding API's constantly makes it not very cost friendly to r&d which is a big draw for this platform.

Is there any way to utilize a custom embeddings model locally at present? and if not, what general steps would I need to go about adding or contributing one?

Additionally, to that I'd love the ability to do the same with LLM's, but that's not as pressing for me as the cost builds up more unpredictably and quickly with calls to embeddings APIs.

Thank you for making this awesome software!

[HenryHengZJ]

Hey thanks for the kind words!

For localized LLMs you can use LocalAI. See how you can use them on Flowise - #123

For embeddings, do you have an inference API endpoint like OpenAI that you can just make the call to it?

[HenryHengZJ]

that's correct. All classes in Flowise follow a similar pattern. Will work on the docs soon for more clarification

[jonmach]

LocalAI is 'in theory' an option, but it is soooooooo very buggy to build, and soooo very fat, and soooo very fragile when running. I managed to pull a standard docker container from dockerhub (34GB!). Building the container manually should have been error free since it was a LocalAI defined Dockerfile, but I gave up in the end and decided to use the fat buggy container. I still couldn't get it to work.

I'm using an alternative LLM server (very fast), and only need embeddings. A local server would also reduce network bandwidth to Huggingface.

I can think of a way to serve them through a FastAPI application that would return a vector, but can't see how I might call this using FlowAI.

Would you have any thoughts on how this could work?

[HenryHengZJ]

try ollama?

[Photize]

especially now its out for Windows, just looking for a good model to run :-)

[jonmach]

Ollama was very buggy, very slow, and very fat. I eventually stumbled over the HuggingFace local embedding server. Really nice, and super fast: https://github.com/huggingface/text-embeddings-inference