Skip to content

attachments-guide.md

Latest commit

 

History

History
294 lines (230 loc) · 7.81 KB

attachments-guide.md

File metadata and controls

294 lines (230 loc) · 7.81 KB
Raw

Multimodal Attachments — Usage Guide

EDDI's attachment pipeline enables multimodal conversations — users can send images, files, and documents alongside text input. Attachments flow through the lifecycle pipeline and are automatically forwarded to vision-capable LLMs.

Quick Start

Send an Image via URL

POST /agents/{conversationId}/say?message=What%20is%20in%20this%20image?
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "url": "https://example.com/photo.png",
      "fileName": "photo.png"
    }
  }
}

Send an Image via Base64

POST /agents/{conversationId}/say?message=Describe%20this%20icon
Content-Type: application/json

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAE...",
      "fileName": "icon.png"
    }
  }
}

The image is automatically forwarded to the LLM as multimodal content. The LLM "sees" the image alongside the text message.

How It Works

Client sends context with attachment_* keys
           │
           ▼
┌──────────────────────────────────┐
│  Conversation.prepareLifecycleData()  │
│                                       │
│  AttachmentContextExtractor parses    │
│  attachment_0, attachment_1, ...      │
│  into List<Attachment> objects        │
│                                       │
│  Stored in memory: "attachments"      │
└──────────────┬───────────────────┘
               │
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────────┐
│BehaviorRules│ │LlmTask│ │Other Tasks │
│             │ │       │ │            │
│ContentType- │ │Multi- │ │Read from   │
│Matcher      │ │modal  │ │memory key  │
│condition    │ │Message│ │"attachments"│
│             │ │Enhancer││            │
└────────┘ └────────┘ └────────────┘

Pipeline Stages

  1. Context Extraction — AttachmentContextExtractor parses attachment_* context keys into Attachment objects
  2. Memory Storage — Attachments are stored as List<Attachment> in the attachments memory key
  3. Rule Matching — ContentTypeMatcher condition matches on MIME types for routing
  4. LLM Forwarding — MultimodalMessageEnhancer converts attachments to langchain4j ImageContent and enhances the user message

Input Paths

Path A: URL Reference (Recommended)

Best for images already hosted somewhere. The LLM provider fetches the image directly from the URL.

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/jpeg",
      "url": "https://cdn.example.com/photos/sunset.jpg",
      "fileName": "sunset.jpg"
    }
  }
}

Path B: Base64 Inline

Best for small images (< 5MB). Data is sent inline as a base64-encoded string.

{
  "attachment_0": {
    "type": "object",
    "value": {
      "mimeType": "image/png",
      "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk...",
      "fileName": "icon.png"
    }
  }
}

Note: base64Data is transient — it's never persisted to MongoDB. For large files, use the upload endpoint (Path C below) or URL references.

Path C: File Upload

For large files, upload them to the storage backend and receive a storage reference:

POST /conversations/{conversationId}/attachments
Content-Type: multipart/form-data

# Form field: file (the binary file)

Response (201):

{
  "storageRef": "gridfs://68abc123def456",
  "fileName": "report.pdf",
  "mimeType": "application/pdf",
  "sizeBytes": 524288
}

The returned storageRef can then be used in subsequent conversation turns by setting it as the url in an attachment context key. The storage backend (GridFS or PostgreSQL) is selected automatically based on the configured datastore.

Response Code Meaning
201 File stored successfully
400 No file provided
503 No attachment storage configured

Context Key Format

Attachment context keys must match the pattern attachment_*:

Key Valid?
attachment_0 ✅
attachment_screenshot ✅
attachment_ ✅
image_0 ❌ (wrong prefix)
attachment ❌ (no suffix)

Required Fields

Field Required Description
mimeType Yes MIME type (e.g., image/png, application/pdf)
url One of url/data External URL reference
data One of url/data Base64-encoded content
fileName No Original filename (for metadata/logging)

If both url and data are present, url takes precedence.

Multiple Attachments

Send multiple attachments by incrementing the key index:

{
  "attachment_0": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page1.png" }
  },
  "attachment_1": {
    "type": "object",
    "value": { "mimeType": "image/png", "url": "https://example.com/page2.png" }
  }
}

All attachments are forwarded to the LLM in a single multimodal user message.

LLM Multimodal Support

The MultimodalMessageEnhancer automatically converts attachments to the appropriate langchain4j content type:

MIME Type langchain4j Content Provider Support
image/* ImageContent OpenAI GPT-4o, Gemini, Claude 3, Ollama (LLaVA)
application/pdf Metadata text (future: PdfFileContent) Gemini
audio/* Metadata text (future: AudioContent) Gemini
Other Metadata text description All (text-only)

For unsupported MIME types, a text description is injected so the LLM knows an attachment was present:

[Attachment: report.csv (text/csv, 15240 bytes)]

Routing with Behavior Rules

Use contentTypeMatcher to create different workflows based on attachment type:

Route Images to Vision Agent

{
  "name": "Image received",
  "actions": ["analyze_image"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "1"
      }
    }
  ]
}

Route PDFs to Document Processor

{
  "name": "Document received",
  "actions": ["process_document"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "application/pdf",
        "minCount": "1"
      }
    }
  ]
}

Require Specific Attachment Count

{
  "name": "Comparison ready",
  "actions": ["compare_images"],
  "conditions": [
    {
      "type": "contentTypeMatcher",
      "configs": {
        "mimeType": "image/*",
        "minCount": "2"
      }
    }
  ]
}

Template Access

Attachments are available in templates via the memory namespace:

Current step attachments: {memory.current.attachments}

This can be useful for logging, debugging, or constructing custom prompts that reference attachment metadata.

Architecture Notes

  • No inline storage: Attachment payloads are never stored inline in conversation memory documents. Only metadata references are persisted.
  • Transient base64: The base64Data field is transient — it exists only during the pipeline turn. For persistence, use the upload endpoint with IAttachmentStorage.
  • DB-agnostic: The IAttachmentStorage SPI supports MongoDB (GridFS) and PostgreSQL (bytea) implementations.
  • GDPR cleanup: IAttachmentStorage.deleteByConversation() removes all attachments when a conversation is deleted.