|
| 1 | +# Milvus Storage Module for vCon Server |
| 2 | + |
| 3 | +This storage module enables storing vCons in a Milvus vector database for semantic search capabilities. |
| 4 | + |
| 5 | +## Description |
| 6 | + |
| 7 | +The Milvus storage module: |
| 8 | +- Extracts text content from vCons (transcripts, summaries, metadata, etc.) |
| 9 | +- Generates embeddings using OpenAI's embedding models |
| 10 | +- Stores the embeddings in Milvus along with related metadata |
| 11 | +- Enables semantic search across vCon content |
| 12 | + |
| 13 | +## Requirements |
| 14 | + |
| 15 | +- Milvus 2.0+ running and accessible |
| 16 | +- OpenAI API key for generating embeddings |
| 17 | +- Additional Python packages: |
| 18 | + ``` |
| 19 | + pymilvus>=2.3.0 |
| 20 | + openai>=1.0.0 |
| 21 | + ``` |
| 22 | + |
| 23 | +## Configuration |
| 24 | + |
| 25 | +Add the following to your config.yml file: |
| 26 | + |
| 27 | +```yaml |
| 28 | +storages: |
| 29 | + milvus: |
| 30 | + module: storage.milvus |
| 31 | + options: |
| 32 | + # Connection settings |
| 33 | + host: "localhost" # Milvus server host |
| 34 | + port: "19530" # Milvus server port |
| 35 | + collection_name: "vcons" # Name of the collection in Milvus |
| 36 | + |
| 37 | + # Embedding settings |
| 38 | + embedding_model: "text-embedding-3-small" # OpenAI embedding model |
| 39 | + embedding_dim: 1536 # Dimensions for the chosen model |
| 40 | + api_key: "your-openai-api-key" # Your OpenAI API key |
| 41 | + organization: "your-org-id" # Optional: Your OpenAI organization ID |
| 42 | + |
| 43 | + # Operation settings |
| 44 | + create_collection_if_missing: true # Whether to create collection if it doesn't exist |
| 45 | + skip_if_exists: true # Skip storing vCons that already exist |
| 46 | + |
| 47 | + # Vector index settings (optional, shown are defaults) |
| 48 | + index_type: "IVF_FLAT" # Vector index type |
| 49 | + metric_type: "L2" # Distance metric type |
| 50 | + nlist: 128 # Number of clusters for IVF indexes |
| 51 | +``` |
| 52 | +
|
| 53 | +### Vector Index Types |
| 54 | +
|
| 55 | +The module supports different vector index types with appropriate parameters: |
| 56 | +
|
| 57 | +#### IVF_FLAT (Default) |
| 58 | +Good balance of search accuracy and speed. Uses more storage but gives exact results within each cluster. |
| 59 | +
|
| 60 | +```yaml |
| 61 | +index_type: "IVF_FLAT" |
| 62 | +metric_type: "L2" # Or "IP" for inner product, or "COSINE" |
| 63 | +nlist: 128 # Number of clusters, higher values = faster search but less accurate |
| 64 | +``` |
| 65 | +
|
| 66 | +#### IVF_SQ8 |
| 67 | +Similar to IVF_FLAT but with scalar quantization (8-bit) to reduce memory usage. Good for large datasets. |
| 68 | +
|
| 69 | +```yaml |
| 70 | +index_type: "IVF_SQ8" |
| 71 | +metric_type: "L2" |
| 72 | +nlist: 128 |
| 73 | +``` |
| 74 | +
|
| 75 | +#### IVF_PQ |
| 76 | +Product Quantization for maximum memory optimization. Sacrifices some accuracy for much smaller index size. |
| 77 | +
|
| 78 | +```yaml |
| 79 | +index_type: "IVF_PQ" |
| 80 | +metric_type: "L2" |
| 81 | +nlist: 128 |
| 82 | +pq_m: 8 # Number of sub-quantizers |
| 83 | +pq_nbits: 8 # Bit depth per quantizer |
| 84 | +``` |
| 85 | +
|
| 86 | +#### HNSW |
| 87 | +Hierarchical Navigable Small World graph index. Very fast for searching, especially with smaller datasets. |
| 88 | +
|
| 89 | +```yaml |
| 90 | +index_type: "HNSW" |
| 91 | +metric_type: "L2" |
| 92 | +m: 16 # Number of edges per node |
| 93 | +ef_construction: 200 # Size of the dynamic candidate list during construction |
| 94 | +``` |
| 95 | +
|
| 96 | +#### FLAT |
| 97 | +The simplest index that compares to every vector. Most accurate but slowest for large datasets. |
| 98 | +
|
| 99 | +```yaml |
| 100 | +index_type: "FLAT" |
| 101 | +metric_type: "L2" |
| 102 | +``` |
| 103 | +
|
| 104 | +### Distance Metrics |
| 105 | +
|
| 106 | +- `L2`: Euclidean distance (default). Good for most embeddings including OpenAI embeddings. |
| 107 | +- `IP`: Inner product. Use when vectors are normalized and you want to measure closeness. |
| 108 | +- `COSINE`: Cosine similarity. Good for measuring the angle between vectors regardless of magnitude. |
| 109 | + |
| 110 | +## Searching vCons in Milvus |
| 111 | + |
| 112 | +While not part of the storage module itself, you can search vCons in Milvus using the pymilvus client: |
| 113 | + |
| 114 | +```python |
| 115 | +from pymilvus import connections, Collection |
| 116 | +
|
| 117 | +# Connect to Milvus |
| 118 | +connections.connect(host="localhost", port="19530") |
| 119 | +
|
| 120 | +# Load collection |
| 121 | +collection = Collection("vcons") |
| 122 | +collection.load() |
| 123 | +
|
| 124 | +# Get embedding for search query (using same OpenAI client as storage module) |
| 125 | +from openai import OpenAI |
| 126 | +client = OpenAI(api_key="your-api-key") |
| 127 | +response = client.embeddings.create(input="your search query", model="text-embedding-3-small") |
| 128 | +query_embedding = response.data[0].embedding |
| 129 | +
|
| 130 | +# Search parameters |
| 131 | +search_params = { |
| 132 | + "metric_type": "L2", |
| 133 | + "params": {"nprobe": 10} |
| 134 | +} |
| 135 | +
|
| 136 | +# Search vCons |
| 137 | +results = collection.search( |
| 138 | + data=[query_embedding], |
| 139 | + anns_field="embedding", |
| 140 | + param=search_params, |
| 141 | + limit=5, |
| 142 | + output_fields=["vcon_uuid", "party_id", "text", "metadata_title"] |
| 143 | +) |
| 144 | +
|
| 145 | +# Process results |
| 146 | +for hits in results: |
| 147 | + for hit in hits: |
| 148 | + print(f"vCon UUID: {hit.entity.get('vcon_uuid')}") |
| 149 | + print(f"Score: {hit.score}") |
| 150 | + print(f"Title: {hit.entity.get('metadata_title')}") |
| 151 | +``` |
| 152 | + |
| 153 | +## Notes |
| 154 | + |
| 155 | +- Vector dimensions must match the embedding model used (1536 for text-embedding-3-small) |
| 156 | +- For larger vCon datasets, consider configuring Milvus index parameters for better performance |
| 157 | +- Milvus collections created by this module include a L2 vector index for similarity search |
0 commit comments