[Features]

[xiaolinyezi]

Do we have plan to support image answer? In the parsing stage, using minio to store image in pdf or doc file. Then correlate image url with text trunks. Finally, in the chat answer stage, show the original image in the chat answer window.

[earayu]

✅ Image Support in ApeRAG

1. Image Storage (MinIO/S3)

During the parsing stage, images from PDF or DOC files are automatically extracted and stored in object storage:

• Supports both MinIO/S3 and local file storage (configured via OBJECT_STORE_TYPE in environment variables)
• Images are stored as binary assets with unique asset_id (MD5 hash)
• Storage path: {document_base_path}/assets/{asset_id}

Reference:

                    metadata = part.metadata.copy()
                    metadata["collection_id"] = collection.id
                    metadata["document_id"] = document_id
                    metadata["source"] = metadata.get("name", "")
                    metadata["asset_id"] = part.asset_id
                    metadata["mimetype"] = mime_type
                    metadata["indexer"] = "vision"
                    metadata["index_method"] = "multimodal_embedding"

2. Image Indexing (Vision Index)

Images are indexed in two ways for semantic search:

Path A: Multimodal Embedding (lines 100-135 in aperag/index/vision_index.py)

• Uses multimodal embedding models (e.g., OpenAI CLIP) to directly convert images to vectors
• Text and images share the same vector space for semantic search

Path B: Vision-to-Text (lines 138-238 in aperag/index/vision_index.py)

• Uses Vision LLM to generate detailed text descriptions of images
• Then converts descriptions to vectors using text embedding models
• Includes comprehensive analysis: text extraction, chart analysis, object recognition

3. Image Correlation with Text

The system maintains complete metadata linking images to their source documents:

        if m.get("asset_id") and m.get("document_id") and m.get("collection_id") and m.get("mimetype"):
            asset_url = f"asset://{m['asset_id']}?document_id={m['document_id']}&collection_id={m['collection_id']}&mime_type={m['mimetype']}"

4. Image Display in Chat Answers

In the chat answer stage, original images are displayed using a special asset:// URL scheme:

Backend (lines 264-293 in aperag/flow/runners/llm.py):

• Retrieves image binary data from object storage
• Converts to base64 data URI for LLM processing
• Includes images in the response references

Frontend (lines 165-186 in web/src/components/markdown.tsx):

• Automatically detects and renders asset:// URLs in markdown
• Fetches and displays the original images from object storage

How to Enable

1. Configure Vision Index when creating a collection:

   • Enable enable_vision in collection config
   • Configure a multimodal embedding model (e.g., text-embedding-3-large) OR a vision LLM (e.g., gpt-4o, claude-3.5-sonnet)

2. Object Storage is configured via environment variables:

   OBJECT_STORE_TYPE=s3  # or "local"
   OBJECT_STORE_S3_ENDPOINT=http://127.0.0.1:9000
   OBJECT_STORE_S3_BUCKET=aperag

The feature has been production-ready and handles the complete workflow you described: extract → store → correlate → display! 🎉