feat(pipeline): ottimizzazione completa con dynamic batching, hash cache O(1) e Tika server obbligatorio

Unificato Dockerfile rimuovendo varianti tika, ora Tika server sempre attivo all'avvio container eliminando cold start 5-10s per file, implementato batching dinamico basato su token budget invece di batch size fisso riducendo chiamate API da 334 a 50-100 per 1000 chunk, aggiunto index file_hash su Qdrant per query O(1) invece di scroll O(N) con cache in-memory per sessione

Author: strawberry-code

CHANGELOG.md

@@ -7,6 +7,36 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [2.0.0] - 2025-12-12
+
+### Breaking Changes
+- **Unica immagine Docker**: rimossi `Dockerfile.tika` e `Dockerfile.tika.local`, ora esiste solo `Dockerfile` con Tika integrato
+- **Tika sempre obbligatorio**: rimosso flag `--no-tika` e logica opzionale, Tika server sempre attivo
+- **Tag immagine semplificato**: usare `ghcr.io/strawberry-code/ragify:latest` (rimosso suffisso `-tika`)
+
+### Added
+- **Dynamic batching**: nuovo sistema di batching basato su token budget invece di batch size fisso
+- **EMBEDDING_TOKEN_BUDGET**: nuova env var (default 1800) per controllare token massimi per batch
+- **Index file_hash**: creazione automatica index su Qdrant per query O(1) invece di scroll O(N)
+- **FileHashCache**: cache in-memory per evitare query ripetute durante indicizzazione
+- **Tika server mode**: Tika avviato come server all'avvio container (porta 9998), elimina cold start 5-10s per file
+
+### Changed
+- `EMBEDDING_BATCH_SIZE` default aumentato da 3 a 20 (ora funziona con token budget)
+- Health check verifica anche Tika server oltre a API, Ollama e Qdrant
+- `check_file_hash_in_qdrant()` usa `count()` O(1) invece di `scroll()` O(N)
+- Entrypoint avvia Qdrant → Ollama → Tika → API in sequenza
+
+### Removed
+- `Dockerfile.tika` e `Dockerfile.tika.local`
+- Flag `--no-tika` e `--non-interactive` da CLI
+- Logica condizionale `use_tika` in pipeline e API
+
+### Performance
+- Riduzione chiamate embedding API: ~334 → ~50-100 per 1000 chunk
+- Eliminato cold start Tika: 5-10s → 0s per file
+- Hash check O(1) con index invece di O(N) scroll
+
 ## [1.3.2] - 2025-12-04
 
 ### Fixed

Dockerfile

@@ -1,4 +1,5 @@
 # Ragify All-in-One Container
+# Includes Ollama, Qdrant, Apache Tika, and Python API
 # Multi-stage build for Docker/Podman
 
 # ============================================
@@ -26,16 +27,24 @@ RUN pip install --no-cache-dir --upgrade pip && \
 FROM python:3.12-slim
 
 LABEL maintainer="Ragify"
-LABEL description="All-in-one RAG documentation search with Ollama, Qdrant, and MCP"
-LABEL version="1.0.0"
+LABEL description="All-in-one RAG documentation search with Ollama, Qdrant, MCP, and Apache Tika"
+LABEL version="2.0.0"
 
-# Install runtime dependencies
+# Install runtime dependencies including Java for Tika server
 RUN apt-get update && apt-get install -y --no-install-recommends \
     tini \
     curl \
     ca-certificates \
+    openjdk-21-jre-headless \
     && rm -rf /var/lib/apt/lists/*
 
+# Set Java environment (create symlink to avoid architecture-specific path)
+RUN ln -s /usr/lib/jvm/java-21-openjdk-* /usr/lib/jvm/java-21
+ENV JAVA_HOME=/usr/lib/jvm/java-21
+ENV PATH="${JAVA_HOME}/bin:${PATH}"
+# Tika JAR path (used by entrypoint.sh to start server)
+ENV TIKA_JAR_PATH=/tmp/tika-server.jar
+
 # Install Ollama - pinned to v0.11.0 to avoid embedding bugs in 0.12.x/0.13.x
 # See: https://github.com/ollama/ollama/issues/13054
 ENV OLLAMA_VERSION=0.11.0
@@ -78,13 +87,40 @@ COPY docker/ ./docker/
 # Make scripts executable
 RUN chmod +x /app/docker/*.sh
 
-# Pre-pull default Ollama model (makes container larger but faster startup)
-# This runs ollama serve temporarily to pull the model
+# Pre-pull default Ollama model
 RUN ollama serve & \
     sleep 5 && \
     ollama pull nomic-embed-text && \
     pkill ollama || true
 
+# Pre-download Tika JAR and ensure it's in the expected location
+RUN python3 <<'PYEOF'
+import os
+import glob
+from tika import parser
+
+# Trigger download (this downloads JAR to /tmp/)
+parser.from_buffer(b'test', xmlContent=False)
+
+# Find the JAR file (handles versioned names like tika-server-standard-3.1.0.jar)
+expected_path = '/tmp/tika-server.jar'
+if os.path.exists(expected_path):
+    print(f'Tika JAR found at {expected_path}')
+else:
+    # Search for versioned JAR
+    jars = glob.glob('/tmp/tika-server*.jar')
+    if jars:
+        print(f'Found JAR: {jars[0]}')
+        os.symlink(jars[0], expected_path)
+        print(f'Created symlink: {expected_path} -> {jars[0]}')
+    else:
+        raise Exception('No Tika JAR found in /tmp/')
+
+# Final verification
+assert os.path.exists(expected_path), f'Tika JAR not found at {expected_path}'
+print(f'Tika JAR verified at {expected_path}')
+PYEOF
+
 # Create data directory for Qdrant
 RUN mkdir -p /data/qdrant
 
@@ -100,7 +136,9 @@ ENV PYTHONUNBUFFERED=1 \
     # Qdrant (internal)
     QDRANT_URL=http://localhost:6333 \
     QDRANT_PATH=/data/qdrant \
-    # Auth (must be provided)
+    # Tika server (internal - started by entrypoint.sh)
+    TIKA_SERVER_ENDPOINT=http://localhost:9998 \
+    # Auth (must be provided for production)
     AUTH_CONFIG="" \
     GITHUB_CLIENT_ID="" \
     GITHUB_CLIENT_SECRET="" \
@@ -112,12 +150,12 @@ VOLUME ["/data"]
 # Expose ports
 EXPOSE 8080 6666
 
-# Health check
-HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+# Health check (verifies API, Ollama, Qdrant, Tika)
+HEALTHCHECK --interval=30s --timeout=10s --start-period=90s --retries=3 \
     CMD /app/docker/healthcheck.sh
 
 # Use tini as init
 ENTRYPOINT ["/usr/bin/tini", "--"]
 
-# Run entrypoint script
+# Run entrypoint script (starts Qdrant, Ollama, Tika, API)
 CMD ["/app/docker/entrypoint.sh"]