Skip to content

RAG

Jump to bottom
Jdaie edited this page Jan 29, 2026 · 14 revisions

I've implemented a lightweight Retrieval-Augmented Generation (RAG) system that enhances responses with relevant context from your knowledge base. This system can run entirely on your Raspberry Pi 5, combining vector search with embedding models to deliver more accurate, context-aware answers.

Prerequisites

List of prerequisites to set up the RAG system:

  • Docker and Vector database: Qdrant (running in a Docker container)
  • Embedding service: Ollama with model "nomic-embed-text"

The vector database and embedding service can be provided by other alternatives as well, as long as they provide compatible APIs.

Memory Requirements

The RAG system requires additional memory for running the embedding model and vector database. Raspberry Pi 5 with >=8GB RAM is recommended for optimal performance.

Use the latest ollama version to avoid this bug: Ollama evicts previously loaded model, although system memory is suficient

Installing Docker and Qdrant

Follow the instructions in the Docker installation guide to install Docker on your Raspberry Pi 5.

After installing Docker, you need to add your user to the docker group, assuming your username is pi:

sudo usermod -aG docker pi

Log out and log back in for the group change to take effect.

After installing Docker, create a folder for Qdrant container:

mkdir -p ~/qdrant
cd ~/qdrant

Create a docker-compose.yml file with the following content:

version: '3.8'
services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    ports:
      - "6333:6333"
    volumes:
      - ./qdrant:/qdrant/storage
    restart: unless-stopped

Change Rpi OS Page size:

Append kernel=kernel8.img to /boot/firmware/config.txt and reboot; this changes the kernel page size to 4k. fluent-bit should be able to run without issues afterwards.

More Detail: https://github.com/fluent/fluent-bit/issues/9730#issuecomment-2705240923

Start the Qdrant container:

cd ~/qdrant
docker compose up -d

Check if Qdrant is running:

curl http://localhost:6333/collections

If everything is set up correctly, you should see an empty collections list:

{
  "result": [],
  "status": "ok"
}

Setting up Ollama with nomic-embed-text

Follow the instructions in the Ollama installation guide to install Ollama on your Raspberry Pi 5.

If you have already running offline version of whisplay chatbot, you can skip the Ollama installation.

Download the "nomic-embed-text" model:

ollama pull nomic-embed-text

Configuring Whisplay AI Chatbot to use RAG

In the .env file of Whisplay AI Chatbot, set the following variables:

## RAG Settings
# Enable RAG (Retrieval-Augmented Generation) by setting the following variable to true
ENABLE_RAG=true

# specify the embedding server and vector database server to use for RAG, default embedding server is ollama
# options for EMBEDDING_SERVER: ollama
EMBEDDING_SERVER=ollama

# specify the vector database server to use for RAG, default vector database server is qdrant
# options for VECTOR_DB_SERVER: qdrant
VECTOR_DB_SERVER=qdrant

# knowledge score threshold for RAG, default is 0.65, score range is 0.0 - 1.0, under this threshold will be considered as no relevant knowledge found
# RAG_KNOWLEDGE_SCORE_THRESHOLD=0.65

## Qdrant for RAG Vector Database
# if you are using Qdrant as the vector database server for RAG, please set the following environment variables, the default host is http://localhost:6333
# QDRANT_HOST=http://localhost:6333

Preparing Knowledge Base

To prepare your knowledge base for RAG, you need to create a collection in Qdrant and populate it with your documents.

Create a folder knowledge in the whisplay-ai-chatbot directory, and put your text documents (in .txt or .md format) into the knowledge folder.

Run the following command to process the documents and populate the Qdrant collection:

bash index_knowledge.sh

The script will read the documents in the knowledge folder, generate embeddings using the "nomic-embed-text" model, and store them in a Qdrant collection named whisplay_knowledge.

The text documents will be split into smaller chunks (~500 characters each) for better retrieval performance.

Using RAG in Whisplay AI Chatbot

When you start the Whisplay AI Chatbot with RAG enabled, it will automatically retrieve relevant knowledge from the Qdrant vector database based on the user's query and provide context-aware responses.

For example, the prepared documents

Raspberry Pi 5 is the latest version of the popular single-board computer developed by the Raspberry Pi Foundation. It features a powerful ARM Cortex-A76 CPU, up to 8GB of RAM, and support for dual 4K displays. The Raspberry Pi 5 is designed for a wide range of applications, including education, hobbyist projects, and professional use cases.

When a user asks about "Raspberry Pi 5", the RAG system will retrieve the relevant knowledge chunk from the Qdrant database and include it in the prompt to the LLM for generating a response.

The workflow is as follows:

  1. User sends a query to the chatbot. ex: "What is Raspberry Pi 5?"
  2. The chatbot generates an embedding for the query using the "nomic-embed-text" model.
  3. The chatbot queries the Qdrant vector database to find relevant knowledge chunks based on the query embedding.
  4. If relevant knowledge is found (based on the knowledge score threshold, select the top relevant chunks), the chatbot includes the knowledge in the prompt to the LLM for generating a response.
  5. The LLM generates a response based on the user's query and the retrieved knowledge, and sends it back to the user.
user: "What is Raspberry Pi 5?"
system: "Answer based on the following knowledge: [retrieved knowledge chunks]"
assistant: "Raspberry Pi 5 is the latest version of the popular single-board computer developed by the Raspberry Pi Foundation..."

Clone this wiki locally