Support for local (self-hosted) LLMs

[danielnbalasoiu]

Hi,

The project is very interesting and I love the idea of being able to ask questions about your data.
For more privacy focused people, is it possible to add support for local LLMs? Maybe Ollama support?

Thank you and keep you the great work you're doing! 👏

[MODSetter]

Hi @danielnbalasoiu
In case you want to test with Local LLMs You can change the llm in backend code to https://python.langchain.com/v0.2/docs/integrations/providers/ollama/

It should be 1 liner change.

I started this project with Local Models but I faced issues with LLMGraphTransformer when using local models.

I will try to work on local llms..... lets see hopefully in a week or two I start getting good responses 🤞

[danielnbalasoiu]

Hi @MODSetter,

Thanks for getting back to me.
When you're saying "change the llm in backend code", how the change should look like?
Change this to point to Ollama's endpoint?

[MODSetter]

Hi @MODSetter,

Thanks for getting back to me. When you're saying "change the llm in backend code", how the change should look like? Change this to point to Ollama's endpoint?

Hi @danielnbalasoiu
Ignore graph_config & llm_config. These fields doesn't store anything ATM.

Just go to server.py & change all llm (

SurfSense/backend/server.py

Line 67 in 789b483

llm = ChatOpenAI(

)

from openai to Ollama like https://python.langchain.com/v0.2/docs/integrations/llms/ollama/

[danielnbalasoiu]

I'll try it. Thanks!

[MODSetter]

@danielnbalasoiu
SurfSense supports local LLM's now 👍