|
1 | | -- [x] Upload Multiple Files |
2 | | -- [x] implement queue worker |
3 | | -- [x] read pdf and chunk the text |
4 | | -- [x] combine those knowledge |
5 | | -- [x] store in vector db |
| 1 | +# Insight Engine |
| 2 | + |
| 3 | +This is the backend for **Insight Engine**, an intelligent application designed to transform a collection of documents into a conversational knowledge base. It allows a user to upload documents (e.g., PDFs), and then ask questions in natural language to receive AI-powered answers based on the information contained within those documents. |
| 4 | + |
| 5 | +The system is built as a multi-container application orchestrated with Docker Compose, ensuring a clean separation of concerns and a production-ready setup. |
| 6 | + |
| 7 | +## System Architecture |
| 8 | + |
| 9 | +The application operates using a **Retrieval-Augmented Generation (RAG)** architecture. |
| 10 | + |
| 11 | +- **API Server (`app`)**: An Express.js server that handles incoming requests. It manages file uploads, adds processing jobs to the queue, and exposes the chat endpoint to the user. |
| 12 | +- **Worker (`worker`)**: A background service using BullMQ that processes long-running tasks. It listens for jobs from the queue, extracts text from documents, generates vector embeddings using the Gemini API, and stores them in ChromaDB. |
| 13 | +- **Redis (`redis`)**: Acts as a high-speed message broker for the BullMQ task queue, facilitating communication between the API Server and the Worker. |
| 14 | +- **ChromaDB (`chroma`)**: The vector database that stores the document embeddings. It enables efficient semantic search to find information relevant to a user's query. |
| 15 | +- **Shared Volume (`uploads`)**: A Docker volume mounted to both the API Server and the Worker, allowing the server to save an uploaded file and the worker to access it for processing. |
| 16 | + |
| 17 | +### How it Works (Step-by-Step) |
| 18 | + |
| 19 | +1. **Knowledge Ingestion**: |
| 20 | + |
| 21 | + - A user uploads PDF files to the `POST /api/v1/pdf/uploads` endpoint. |
| 22 | + - The API Server clears the old ChromaDB collection to start a fresh session. |
| 23 | + - It adds a job for each file to the Redis queue. |
| 24 | + - The Worker picks up each job, processes the PDF, creates chunks, generates embeddings with the Gemini API, and stores them in ChromaDB. |
| 25 | + - The worker then deletes the temporary file from the shared volume. |
| 26 | + |
| 27 | +2. **Knowledge Retrieval**: |
| 28 | + - A user sends a question to the `POST /api/v1/pdf/chat` endpoint. |
| 29 | + - The API Server queries ChromaDB to retrieve the most relevant text chunks based on the question's meaning. |
| 30 | + - It _augments_ a prompt by combining this retrieved context with the original question. |
| 31 | + - This detailed prompt is sent to the Gemini generative model. |
| 32 | + - The model _generates_ a final answer based on the provided context, which is then sent back to the user. |
| 33 | + |
| 34 | +## Project Setup & How to Run |
| 35 | + |
| 36 | +1. **Clone the Repository** |
| 37 | + |
| 38 | + ```bash |
| 39 | + git clone <your-repo-url> |
| 40 | + cd insight-engine |
| 41 | + ``` |
| 42 | + |
| 43 | +2. **Create Environment File** |
| 44 | + Create a `.env` file in the root directory and add your configuration. |
| 45 | + |
| 46 | + ```env |
| 47 | + # Server Port |
| 48 | + PORT=4000 |
| 49 | +
|
| 50 | + # Redis Configuration for Docker |
| 51 | + REDIS_HOST=redis |
| 52 | + REDIS_PORT=6379 |
| 53 | +
|
| 54 | + # ChromaDB Configuration for Docker |
| 55 | + CHROMA_HOST=chroma |
| 56 | + CHROMA_PORT=8000 |
| 57 | +
|
| 58 | + # Your Gemini API Key |
| 59 | + GEMINI_API_KEY=AIzaSy... |
| 60 | + ``` |
| 61 | + |
| 62 | +3. **Run with Docker Compose** |
| 63 | + Make sure you have Docker Desktop running. Then, from the project root, run: |
| 64 | + |
| 65 | + ```bash |
| 66 | + docker-compose up --build |
| 67 | + ``` |
| 68 | + |
| 69 | + This command will build the Docker images and start all the necessary services (`app`, `worker`, `redis`, `chroma`). |
| 70 | + |
| 71 | +4. **Interact with the API** |
| 72 | + - **Upload Files:** Send a `POST` request with `form-data` to `http://localhost:4000/api/v1/pdf/uploads`. The key for your files should be `files`. |
| 73 | + - **Chat:** Send a `POST` request with a JSON body to `http://localhost:4000/api/v1/pdf/chat`. The body should look like: `{ "question": "your question here" }`. |
| 74 | + |
| 75 | +## Development |
| 76 | + |
| 77 | +For local development with hot-reloading, create a `docker-compose.override.yml` file to mount your source code into the containers and use `nodemon` (as configured in your `package.json`). This allows changes to your code to be reflected instantly without rebuilding the image. |
| 78 | + |
| 79 | +## Architecture Diagram |
| 80 | + |
| 81 | +```mermaid |
| 82 | +graph TD |
| 83 | + subgraph "User / Client" |
| 84 | + User(["<br><b>User</b><br>via Frontend / Postman"]) |
| 85 | + end |
| 86 | +
|
| 87 | + subgraph "Backend Application (Docker)" |
| 88 | + subgraph "API Server (app-1)" |
| 89 | + API[("Express.js API<br>localhost:4000")] |
| 90 | + end |
| 91 | +
|
| 92 | + subgraph "Background Worker (worker-1)" |
| 93 | + Worker[("BullMQ Worker")] |
| 94 | + end |
| 95 | +
|
| 96 | + subgraph "Databases & Services" |
| 97 | + Redis[("Redis<br>Queue")] |
| 98 | + Chroma[("ChromaDB<br>Vector Store")] |
| 99 | + end |
| 100 | +
|
| 101 | + subgraph "Shared Storage" |
| 102 | + Volume[("</br>uploads</br>Shared Volume")] |
| 103 | + end |
| 104 | + end |
| 105 | +
|
| 106 | + subgraph "External Services" |
| 107 | + Gemini[("Google Gemini AI<br>Embeddings & Generation")] |
| 108 | + end |
| 109 | +
|
| 110 | + %% Ingestion Flow (Uploading a document) |
| 111 | + User -- "1. Uploads PDF (POST /uploads)" --> API |
| 112 | + API -- "2. Saves file to" --> Volume |
| 113 | + API -- "3. Enqueues Job" --> Redis |
| 114 | + Worker -- "4. Dequeues Job" --> Redis |
| 115 | + Worker -- "5. Reads file from" --> Volume |
| 116 | + Worker -- "6. Generates Embeddings" --> Gemini |
| 117 | + Worker -- "7. Stores Embeddings" --> Chroma |
| 118 | + Worker -- "8. Deletes file from" --> Volume |
| 119 | +
|
| 120 | + %% Retrieval Flow (Asking a question) |
| 121 | + User -- "9. Asks Question (POST /chat)" --> API |
| 122 | + API -- "10. Retrieves Context" --> Chroma |
| 123 | + API -- "11. Augments Prompt & Generates Answer" --> Gemini |
| 124 | + API -- "12. Sends Answer" --> User |
| 125 | +
|
| 126 | + classDef default fill:#1E293B,stroke:#334155,stroke-width:2px,color:#fff; |
| 127 | + classDef user fill:#2563EB,stroke:#1D4ED8,stroke-width:2px,color:#fff; |
| 128 | + classDef external fill:#166534,stroke:#14532D,stroke-width:2px,color:#fff; |
| 129 | +
|
| 130 | + class User user; |
| 131 | + class Gemini external; |
| 132 | +``` |
0 commit comments