Local LLM RAG System

A Retrieval-Augmented Generation system that allows querying local documents using Ollama and vector embeddings.

Features

• 🔒 Privacy-focused: All processing happens locally
• 🚀 Fast semantic search using FAISS vector database
• 🤖 Multiple LLM support via Ollama (llama3.2, mistral, gemma)
• 📄 Document retrieval with context-aware answers

Tech Stack

• Ollama - Local LLM inference
• LangChain - RAG orchestration
• FAISS - Vector database
• HuggingFace Embeddings - Text embeddings (all-MiniLM-L6-v2)
• Python 3.13

Installation

1. Install Ollama:

# Download from https://ollama.ai
ollama pull llama3.2

2. Install Python dependencies:

pip install langchain-community langchain-core sentence-transformers faiss-cpu

Usage

1. Place your text documents in the docs/ folder

2. Run the RAG system:

python rag_demo.py

3. Ask questions about your documents!

Example

Ask a question about your documents: what is python used for
==================================================
Answer: According to the context, Python is used for:
1. AI
2. Web development
3. Automation

How It Works

1. Documents are loaded and converted to vector embeddings
2. Embeddings stored in FAISS vector database
3. User question is embedded and used for similarity search
4. Retrieved context + question sent to Ollama LLM
5. LLM generates answer based on retrieved context