How can I use a vllm deployed LLM in Flowise?

[benjaminmateev]

Hey Flowise community,

I am experimenting with Flowise as a solution to quickly build LLM flows for our copilot application. We fine-tune open-source models like Mistral and LLama3 and run them on a Kubeflow cluster, via kserve and vllm. I have not been able to find any information on using vLLM in flowise or a node in the nodes overview that could be used for that case.

Is this at all possible? Is there some other node I can use that basically works the same way?

Thank you so much!
Ben

[smpnet74]

As long as you are configuring the vllm.entrypoints to openai and wrap vllm with something like fastapi, you can use the chatlocalAI node to interact.