|
| 1 | +# Building a Multimodal RAG Pipeline with Elasticsearch: The Story of Gotham City |
| 2 | + |
| 3 | +This repository contains the code for implementing a Multimodal Retrieval-Augmented Generation (RAG) system using Elasticsearch. The system processes and analyzes different types of evidence (images, audio, text, and depth maps) to solve a crime in Gotham City. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The pipeline demonstrates how to: |
| 8 | +- Generate unified embeddings for multiple modalities using ImageBind |
| 9 | +- Store and search vectors efficiently in Elasticsearch |
| 10 | +- Analyze evidence using GPT-4 to generate forensic reports |
| 11 | + |
| 12 | +## Prerequisites |
| 13 | + |
| 14 | +- A Docker runtime with 8GB+ free ram |
| 15 | + - GPU is optional, but recommended |
| 16 | +- Elasticsearch cluster (cloud or local) |
| 17 | +- OpenAI API key - Setup an OpenAI account and create a [secret key](https://platform.openai.com/docs/quickstart) |
| 18 | + |
| 19 | +## Quick Start |
| 20 | + |
| 21 | +This example runs four stages as docker compose services: |
| 22 | + |
| 23 | +```mermaid |
| 24 | +graph TD |
| 25 | + verify-file-structure --> generate-embeddings |
| 26 | + generate-embeddings --> index-content |
| 27 | + index-content --> search-and-analyze |
| 28 | +``` |
| 29 | + |
| 30 | +First, copy [env.example](env.example) to `.env` and fill in values noted inside. |
| 31 | + |
| 32 | + |
| 33 | +Edit the `.env` file and fill in your credentials: |
| 34 | + |
| 35 | +```env |
| 36 | +# Elasticsearch Configuration |
| 37 | +ELASTICSEARCH_URL="https://your-elasticsearch-endpoint:443" |
| 38 | +ELASTICSEARCH_API_KEY="your-api-key" |
| 39 | +# If not using API key, uncomment these and fill them in: |
| 40 | +# ELASTICSEARCH_USER=elastic |
| 41 | +# ELASTICSEARCH_PASSWORD=elastic |
| 42 | +
|
| 43 | +# OpenAI Configuration |
| 44 | +OPENAI_API_KEY="your-openai-api-key" |
| 45 | +``` |
| 46 | + |
| 47 | +### 2. Configure Docker Resources |
| 48 | + |
| 49 | +The ImageBind model requires significant memory. Ensure Docker has enough resources: |
| 50 | + |
| 51 | +- **Memory**: At least 8GB (16GB recommended) |
| 52 | +- **Storage**: At least 10GB free space |
| 53 | + |
| 54 | +For Docker Desktop users: |
| 55 | +1. Open Docker Desktop settings |
| 56 | +2. Go to Resources > Advanced |
| 57 | +3. Increase memory allocation to at least 8GB |
| 58 | +4. Apply & Restart |
| 59 | + |
| 60 | +### 3. Running the Complete Pipeline |
| 61 | + |
| 62 | +To run the entire pipeline from file structure verification to evidence analysis: |
| 63 | + |
| 64 | +```bash |
| 65 | +docker compose run --build --rm search-and-analyze |
| 66 | +``` |
| 67 | + |
| 68 | +This command will: |
| 69 | +1. Build the Docker image if needed |
| 70 | +2. Run each stage in sequence |
| 71 | +3. Cache the ImageBind model weights for future runs |
| 72 | + |
| 73 | +The first run will take longer as it builds the image and downloads model weights (~4.5GB). |
| 74 | + |
| 75 | +### 4. Running Individual Stages |
| 76 | + |
| 77 | +If you prefer to run each stage separately: |
| 78 | + |
| 79 | +```bash |
| 80 | +# File structure verification |
| 81 | +docker compose run --build --rm verify-file-structure |
| 82 | + |
| 83 | +# Generate embeddings |
| 84 | +docker compose run --build --rm generate-embeddings |
| 85 | + |
| 86 | +# Index content in Elasticsearch |
| 87 | +docker compose run --build --rm index-content |
| 88 | + |
| 89 | +# Search and analyze evidence |
| 90 | +docker compose run --build --rm search-and-analyze |
| 91 | +``` |
| 92 | + |
| 93 | +To skip dependency checks when running a specific stage: |
| 94 | + |
| 95 | +```bash |
| 96 | +docker compose run --no-deps --build --rm search-and-analyze |
| 97 | +``` |
0 commit comments