Merge pull request #45 from AI4quantum/gliu/semantic-chunking-only

enhancement: integrate semantic chunking

Author: george-lhj

README.md

@@ -6,7 +6,7 @@ A modular vector database interface supporting multiple backends (Weaviate, Milv
 
 - **Multi-backend support**: Weaviate and Milvus vector databases
 - **Flexible embedding strategies**: Support for pre-computed vectors and multiple embedding models
-- **Pluggable document chunking**: None (default), Fixed (size/overlap), Sentence-aware
+- **Pluggable document chunking**: None (default), Fixed (size/overlap), Sentence-aware, Semantic (AI-powered)
 - **Unified API**: Consistent interface across different vector database implementations
 - **Factory pattern**: Easy creation and switching between database types
 - **MCP Server**: Model Context Protocol server for AI agent integration with multi-database support
@@ -18,6 +18,62 @@ A modular vector database interface supporting multiple backends (Weaviate, Milv
 - **Environment variable substitution**: Dynamic configuration with `{{ENV_VAR_NAME}}` syntax
 - **Safety features**: Confirmation prompts for destructive operations with `--force` flag bypass
 
+## Chunking Strategies
+
+Maestro Knowledge supports multiple document chunking strategies to optimize how your documents are split for vector search:
+
+### Available Strategies
+
+- **None**: No chunking performed (default)
+- **Fixed**: Split documents into fixed-size chunks with optional overlap
+- **Sentence**: Split documents at sentence boundaries with size limits  
+- **Semantic**: Identifies semantic boundaries using sentence embeddings
+
+### Semantic Chunking
+
+The semantic chunking strategy uses sentence transformers to intelligently split documents:
+
+```python
+from src.chunking import ChunkingConfig, chunk_text
+
+# Configure semantic chunking
+config = ChunkingConfig(
+    strategy="Semantic",
+    parameters={
+        "chunk_size": 768,      # Default for semantic (vs 512 for others)
+        "overlap": 0,           # Optional overlap between chunks
+        "window_size": 1,       # Context window for similarity calculation
+        "threshold_percentile": 90.0,  # Percentile threshold for splits
+        "model_name": "all-MiniLM-L6-v2"  # Sentence transformer model
+    }
+)
+
+# Chunk your text
+chunks = chunk_text("Your document text here...", config)
+```
+
+**Key Benefits**:
+- Preserves semantic meaning across chunk boundaries
+- Automatically finds natural break points in text
+- Respects size limits while maintaining context
+- Uses 768 character default (optimal for semantic understanding)
+
+**Note**: Semantic chunking uses sentence-transformers for chunking decisions, but the resulting chunks are embedded using your collection's embedding model (e.g., nomic-embed-text) for search operations.
+
+### Testing Semantic Chunking
+
+You can test the semantic chunking functionality using the CLI:
+
+```bash
+# Check collection information to see chunking strategy
+cli/maestro-k collection info --vdb "Qiskit_studio_algo" --name "Qiskit_studio_algo"
+
+# Search with semantic chunking to see results
+./cli/maestro-k search "quantum circuit" --vdb qiskit_studio_algo --collection qiskit_studio_algo --doc-limit 1
+```
+
+**Note**: The semantic chunking strategy uses sentence-transformers for chunking decisions, while the collection's own embedding model is used for search operations.
+
 ## Quick Start
 
 ### Installation

cli/README.md

@@ -9,7 +9,7 @@ A command-line interface for interacting with the Maestro Knowledge MCP server w
 - **List collections**: List all collections in a specific vector database
 - **List documents**: List documents in a specific collection of a vector database
 - **Query documents**: Query documents using natural language with semantic search
-- **Pluggable document chunking**: Configure per-collection chunking (None, Fixed with size/overlap, Sentence)
+- **Pluggable document chunking**: Configure per-collection chunking (None, Fixed with size/overlap, Sentence, Semantic)
    - Discover supported strategies with `maestro-k chunking list`
 - **Create vector databases**: Create vector databases from YAML configuration files
 - **Delete vector databases**: Delete vector databases by name
@@ -317,6 +317,33 @@ Override the MCP server URI via command-line flag:
 ./maestro-k chunking list
 ```
 
+#### Chunking Strategies
+
+**None**: No chunking is performed (default)
+**Fixed**: Split documents into fixed-size chunks with optional overlap
+**Sentence**: Split documents at sentence boundaries with size limits
+**Semantic**: AI-powered chunking that identifies semantic boundaries using sentence embeddings
+
+#### Semantic Chunking Example
+
+Semantic chunking uses sentence transformers to identify natural break points in documents:
+
+```bash
+# Create collection with semantic chunking
+./maestro-k create collection my-database my-collection \
+  --chunking-strategy=Semantic \
+  --chunk-size=768 \
+  --chunk-overlap=0
+
+# The semantic strategy will:
+# - Split text into sentences
+# - Use AI embeddings to find semantic boundaries
+# - Respect the chunk_size limit while preserving meaning
+# - Default to 768 characters (vs 512 for other strategies)
+```
+
+**Note**: Semantic chunking uses sentence-transformers for chunking decisions, but the resulting chunks are embedded using your collection's embedding model (e.g., nomic-embed-text) for search operations.
+
 ### Environment Variable Substitution in YAML Files
 
 The CLI supports environment variable substitution in YAML files using the `{{ENV_VAR_NAME}}` syntax. This allows you to use environment variables directly in your configuration files:
@@ -986,3 +1013,20 @@ go test -v ./tests/...
 ## License
 
 Apache 2.0 License - see the main project LICENSE file for details.
+
+## Semantic Chunking Example
+
+The CLI supports semantic chunking for intelligent document splitting:
+
+```bash
+# Create a collection with semantic chunking
+cli/maestro-k collection create --vdb my-vdb --name my-collection
+
+# Check collection information to see chunking strategy
+cli/maestro-k collection info --vdb "Qiskit_studio_algo" --name "Qiskit_studio_algo"
+
+# Search with semantic chunking to see results
+./cli/maestro-k search "quantum circuit" --vdb qiskit_studio_algo --collection qiskit_studio_algo --doc-limit 1
+```
+
+**Note**: The semantic chunking strategy uses sentence-transformers for chunking decisions, while the collection's own embedding model is used for search operations.

pyproject.toml

@@ -22,6 +22,9 @@ dependencies = [
     "jsonschema>=4.25.0",
     "fastmcp>=2.11.0",
     "six>=1.17.0",
+    "sentence-transformers>=2.5.1",
+    "scikit-learn>=1.5.0",
+    "numpy>=1.26.0",
 ]
 
 [tool.ruff.lint]

src/chunking/__init__.py

@@ -10,11 +10,13 @@
 # Re-export strategy names for discovery if needed
 from .none import none_chunk
 from .sentence import sentence_chunk
+from .semantic_chunking import semantic_chunk
 
 __all__ = [
     "ChunkingConfig",
     "chunk_text",
     "none_chunk",
     "fixed_chunk",
     "sentence_chunk",
+    "semantic_chunk",
 ]

src/chunking/common.py

@@ -47,8 +47,10 @@ def chunk_text(
 
     # apply defaults when strategy is set and parameters missing
     if strategy != "None":
-        # default chunk size 512 and overlap 0
-        params = {"chunk_size": 512, "overlap": 0}
+        if strategy == "Semantic":
+            params = {"chunk_size": 768, "overlap": 0}
+        else:
+            params = {"chunk_size": 512, "overlap": 0}
         params.update(parameters)
     else:
         params = {}