Milvus Vector Database Setup

This guide covers the setup, configuration, and management of the Milvus vector database used by ISC-CodeConnect for storing and retrieving code embeddings.

Overview

ISC-CodeConnect uses Milvus as its vector database to store embedded code chunks from Salesforce repositories. The system maintains 5 specialized collections for different code types, enabling efficient similarity search and retrieval.

Production Environment: The system is configured to use IBM's CIO shared Milvus service, providing a managed, high-performance vector database solution.

Architecture

IBM CIO Milvus Service

ISC-CodeConnect connects to IBM's centrally managed Milvus service through the following architecture:

┌─────────────────────┐    ┌──────────────────────┐    ┌─────────────────────┐
│ ISC-CodeConnect     │    │ IBM CIO Network      │    │ Milvus Service      │
│ Application         │◄──►│ Authentication &     │◄──►│ - Managed Instance  │
│ - Connection Pool   │    │ Access Control       │    │ - High Availability │
│ - Query Router      │    │ - IAM Integration    │    │ - Auto Scaling      │
│ - Health Monitor    │    │ - Network Security   │    │ - Backup/Recovery   │
└─────────────────────┘    └──────────────────────┘    └─────────────────────┘

Connection Management

The system uses a sophisticated connection pooling strategy:

Singleton Manager: MilvusConnectionManager ensures single instance
Connection Pools: 2 connections per collection for load distribution
Round-Robin Distribution: Queries distributed across available connections
Health Monitoring: Automatic connection health checks and recovery

Configuration

Environment Variables

Configure the connection to IBM's Milvus service:

# Enable Milvus vector database
MILVUS_ENABLED=true

# IBM CIO Milvus Service Connection
MILVUS_URI=https://your-milvus-instance.cloud.ibm.com:19530
MILVUS_USERNAME=your_ibm_username
MILVUS_APIKEY=your_api_key_or_token
MILVUS_DB_NAME=isc_codeconnect_db

# Performance Settings
FETCH_K=100
EF=200

Authentication

The system supports multiple authentication methods:

API Key Authentication (Recommended)

MILVUS_USERNAME=your_service_account
MILVUS_APIKEY=your_generated_api_key

IAM Token Authentication

MILVUS_USERNAME=your_ibm_username
MILVUS_APIKEY=your_iam_token

Connection Parameters

The connection is configured with these parameters:

connection_args={
    "uri": settings.MILVUS_URI,
    "user": settings.MILVUS_USERNAME,
    "password": settings.MILVUS_APIKEY,
    "db_name": settings.MILVUS_DB_NAME,
}

Local Development Setup

For local development and testing, you can use a local Milvus instance:

Option 1: Milvus Standalone (Development)

# Download and start Milvus standalone
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh

# Or using Docker directly
docker run -d --name milvus-standalone \
  -p 19530:19530 \
  -p 9091:9091 \
  -v $(pwd)/volumes/milvus:/var/lib/milvus \
  milvusdb/milvus:v2.3.0 \
  milvus run standalone

Option 2: Docker Compose (Development)

Create docker-compose.milvus.yml:

version: "3.5"

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  milvus:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.3.0
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

For local development, configure:

# Local Milvus instance
MILVUS_ENABLED=true
MILVUS_URI=http://localhost:19530
MILVUS_USERNAME=  # Leave empty for local
MILVUS_APIKEY=    # Leave empty for local
MILVUS_DB_NAME=local_dev_db

Collections Structure

ISC-CodeConnect maintains 5 specialized collections in Milvus:

Collection Overview

Collection	Purpose	Document Types	Metadata Fields
`apex_code`	Salesforce Apex classes and methods	Production code	`class_name`, `method_name`, `file_path`
`apex_test_code`	Apex test classes and patterns	Test implementations	`test_class_name`, `test_method_name`, `class_under_test`
`lwc_js_code`	Lightning Web Component JavaScript	Component logic	`component_name`, `method_name`, `lwc_name`
`lwc_html_code`	Lightning Web Component templates	HTML templates	`component_name`, `template_name`, `lwc_name`
`lwc_test_code`	Jest test files for LWC	Frontend tests	`test_file_name`, `component_under_test`, `test_type`

Collection Schema

Each collection uses a consistent schema:

{
    "id": "auto-generated",           # Primary key
    "text": "embedded_content",       # The actual code/text content
    "metadata": {                    # Collection-specific metadata
        "source": "file_path",
        "class_name": "ClassName",   # For Apex collections
        "component_name": "CompName", # For LWC collections
        "method_name": "methodName",
        # ... other metadata fields
    },
    "vector": [0.1, 0.2, ...]       # Embedding vector
}

Indexing Strategy

Each collection is configured with:

Vector Index: HNSW (Hierarchical Navigable Small World) for fast similarity search
Scalar Index: B-tree indexes on metadata fields for filtering
Dynamic Fields: Enabled for flexible metadata expansion

Connection Management

MilvusConnectionManager

The system uses a singleton connection manager:

from src.retrievers.milvus_manager import MilvusConnectionManager

# Initialize during startup
manager = await MilvusConnectionManager.initialize_connections()

# Get connection for specific collection
connection = manager.get_connection("apex_code")

Connection Pooling

The manager maintains connection pools:

Pool Size: 2 connections per collection
Distribution: Round-robin query distribution
Health Monitoring: Automatic connection health checks
Recovery: Automatic reconnection on failures

For detailed information, see Milvus Connection Manager.

Integration with Retriever Factory

The connection manager integrates with the factory pattern:

from src.retrievers.factory import CodeRetrieverFactory

# Factory automatically uses Milvus when enabled
factory = CodeRetrieverFactory()

# Get retriever for specific collection
retriever = factory.get_retriever("apex_code", state=agent_state)

For detailed information, see Retriever Factory.

Performance Optimization

Index Configuration

# HNSW Index Parameters
index_params = {
    "metric_type": "L2",          # Distance metric
    "index_type": "HNSW",         # Index algorithm
    "params": {
        "M": 16,                  # Number of connections
        "efConstruction": 200,    # Search scope during construction
    }
}

# Search Parameters
search_params = {
    "metric_type": "L2",
    "params": {
        "ef": settings.EF,        # Search scope (default: 200)
    }
}

Query Optimization

Batch Processing: Process multiple queries concurrently
Connection Pooling: Distribute load across connections
Caching: State-based caching for repeated queries
Compression: Use FlashRank reranking for relevance

Performance Monitoring

The system tracks query performance:

# Query time tracking
manager.log_query_time(start_time, end_time)

# Average calculation after 100 queries
avg_time = sum(query_times) / len(query_times)
print(f"Average query time: {avg_time:.2f}s")

Security

Authentication

API Key Management: Secure storage of Milvus credentials
Token Rotation: Support for periodic token updates
Network Security: Encrypted connections (HTTPS/TLS)
Access Control: Role-based access through IBM CIO

Data Protection

Encryption: Data encrypted in transit and at rest
Access Logs: Query and access logging
Backup Strategy: Automated backups through IBM CIO
Compliance: Adherence to IBM security standards

Monitoring and Troubleshooting

Health Checks

# Check all connections
healthy = manager.check_all_connections()
if not healthy:
    print("Some connections are unhealthy")

Common Issues

Issue	Symptoms	Solution
Connection Timeout	Slow queries, timeouts	Check network connectivity
Authentication Error	Connection failures	Verify credentials
Collection Not Found	Query errors	Ensure collections exist
Poor Performance	Slow responses	Check index configuration

Debugging

Enable debug logging:

import logging
logging.getLogger("pymilvus").setLevel(logging.DEBUG)

Performance Metrics

Monitor these key metrics:

Query Latency: Average time per query
Connection Health: Percentage of healthy connections
Cache Hit Rate: Effectiveness of caching
Error Rate: Frequency of failed queries

Maintenance

Regular Tasks

Health Monitoring: Daily connection health checks
Performance Review: Weekly query time analysis
Index Optimization: Monthly index performance review
Connection Pool Tuning: Adjust pool sizes based on load

Backup and Recovery

Automated Backups: Handled by IBM CIO Milvus service
Point-in-Time Recovery: Available through service
Disaster Recovery: Multi-region setup for high availability
Data Validation: Regular integrity checks

Migration

From Local to Production

When migrating from local development to IBM CIO production:

Update Configuration:

MILVUS_URI=https://prod-milvus.cloud.ibm.com:19530
MILVUS_USERNAME=prod_service_account
MILVUS_APIKEY=prod_api_key

Data Migration:
- Export collections from local instance
- Import to production instance
- Verify data integrity
Performance Testing:
- Test query performance
- Validate connection pooling
- Monitor resource usage

Version Upgrades

Milvus Updates: Coordinated with IBM CIO team
Client Library: Update pymilvus and langchain-milvus
Schema Changes: Plan for backward compatibility
Testing: Comprehensive testing before deployment

FilesExpand file tree

milvus-setup.md

Latest commit

History

milvus-setup.md

File metadata and controls

Milvus Vector Database Setup

Overview

Architecture

IBM CIO Milvus Service

Connection Management

Configuration

Environment Variables

Authentication

Connection Parameters

Local Development Setup

Option 1: Milvus Standalone (Development)

Option 2: Docker Compose (Development)

Collections Structure

Collection Overview

Collection Schema

Indexing Strategy

Connection Management

MilvusConnectionManager

Connection Pooling

Integration with Retriever Factory

Performance Optimization

Index Configuration

Query Optimization

Performance Monitoring

Security

Authentication

Data Protection

Monitoring and Troubleshooting

Health Checks

Common Issues

Debugging

Performance Metrics

Maintenance

Regular Tasks

Backup and Recovery

Migration

From Local to Production

Version Upgrades

Related Documentation