merge complete

Author: howethomas

README.md

@@ -268,3 +268,44 @@ vcon-server-conserver-1  | {"asctime": "2024-08-23 17:27:22,240", "levelname": "
 ```
 
 The [vCon admin program](https://github.com/vcon-dev/vcon-admin) is a nice tool for managing the conserver. 
+
+## Storage Modules
+
+### Milvus Vector Database Storage
+
+The vcon-server includes support for storing vCons in Milvus, a vector database that enables semantic search across vCon content. This is particularly useful for finding conversations based on meaning rather than exact keyword matches.
+
+To set up Milvus storage:
+
+1. Install the required packages:
+   ```bash
+   poetry add pymilvus>=2.3.0 openai>=1.54.3 python-dateutil
+   ```
+
+2. Add Milvus storage configuration to your config.yml:
+   ```yaml
+   storages:
+     milvus:
+       module: storage.milvus
+       options:
+         host: "localhost"                  # Milvus server host
+         port: "19530"                      # Milvus server port
+         collection_name: "vcons"           # Name of collection in Milvus
+         embedding_model: "text-embedding-3-small"  # OpenAI embedding model
+         embedding_dim: 1536                # Dimensions for the embedding model
+         api_key: "your-openai-api-key"     # Your OpenAI API key
+         organization: "your-org-id"        # Optional: Your OpenAI organization ID
+         create_collection_if_missing: true # Auto-create collection if needed
+   ```
+
+3. Include the Milvus storage in your processing chain:
+   ```yaml
+   chains:
+     main_chain:
+       # ... other configuration ...
+       storages:
+         - milvus
+         # ... other storages ...
+   ```
+
+See the [Milvus Storage Module README](server/storage/milvus/README.md) for more details on configuration and usage.

example_config.yml

@@ -52,6 +52,36 @@ storages:
       aws_access_key_id: some_key
       aws_secret_access_key: some_secret
       aws_bucket: some_bucket
+  milvus:
+    module: storage.milvus
+    options:
+      # Connection settings
+      host: "localhost"
+      port: "19530"
+      collection_name: "vcons"
+      
+      # Embedding settings
+      embedding_model: "text-embedding-3-small"
+      embedding_dim: 1536
+      api_key: "your-openai-api-key"
+      organization: ""
+      
+      # Operation settings
+      create_collection_if_missing: true
+      skip_if_exists: true
+      
+      # Vector index settings (Default: IVF_FLAT with L2 distance)
+      index_type: "IVF_FLAT"  # Options: IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW, FLAT
+      metric_type: "L2"       # Options: L2, IP, COSINE
+      nlist: 128              # For IVF indexes: number of clusters
+      
+      # Advanced HNSW settings (used only if index_type is HNSW)
+      # m: 16                 # Number of edges per node 
+      # ef_construction: 200  # Size of dynamic candidate list during construction
+      
+      # Advanced IVF_PQ settings (used only if index_type is IVF_PQ)
+      # pq_m: 8               # Number of sub-quantizers
+      # pq_nbits: 8           # Bit depth per quantizer
 chains:
   sample_chain:
     links:
@@ -65,6 +95,7 @@ chains:
     - mongo
     - postgres
     - s3
+    - milvus
     egress_lists:
     - test_output
     enabled: 1

poetry.lock

@@ -42,6 +42,8 @@ anyio = "^4.8.0"
 faker = "^33.3.1"
 jq = "^1.8.0"
 pytest-redis = "^3.1.3"
+pymilvus = ">=2.3.0"
+python-dateutil = "^2.8.2"
 
 
 [tool.poetry.group.dev.dependencies]

pyproject.toml

@@ -0,0 +1,82 @@
+# Conserver Installation Script
+
+A bash script to easily install and configure Conserver vCon Server with all necessary components.
+
+## Overview
+
+This script automates the installation process for the Conserver vCon Server, setting up:
+- Docker containers
+- MongoDB
+- Redis
+- vCon Server API
+- Admin interface
+- Traefik for routing
+
+## Quick Installation
+
+```bash
+curl -o install_conserver.sh https://raw.githubusercontent.com/your-username/your-repo/main/install_conserver.sh
+chmod +x install_conserver.sh
+sudo ./install_conserver.sh --domain your-domain.com --email your-email@example.com
+```
+
+## Requirements
+
+- Ubuntu Linux (recommended)
+- Root access
+- Public internet access
+- DNS record pointing to your server for the domain you specify
+
+## Usage Options
+
+```bash
+./install_conserver.sh [options]
+```
+
+### Required Parameters:
+- `-d, --domain DOMAIN` - Domain name for the installation
+- `-e, --email EMAIL` - Email for DNS registration and notifications
+
+### Optional Parameters:
+- `-t, --token TOKEN` - API token (default: randomly generated)
+- `-h, --help` - Display help information
+
+## Post-Installation
+
+After successful installation, you'll have access to:
+
+- **Admin Portal**: `https://your-domain.com/admin/` (default login: admin/admin)
+- **API Documentation**: `https://your-domain.com/api/docs`
+- **API Token**: Displayed at the end of installation (save this!)
+- **MongoDB Express**: `https://your-domain.com/mongo-express`
+- **Redis Stack**: `http://localhost:8001`
+
+## Configuration
+
+The script creates the following configuration:
+- Docker network named "conserver"
+- MongoDB for vCon storage
+- Redis for message queuing
+- Pipeline configuration with tags set to "conserver"
+
+## Troubleshooting
+
+If you encounter issues:
+
+1. Check Docker container status: `docker ps`
+2. Inspect Docker network: `docker network inspect conserver`
+3. View container logs: `docker logs [container-name]`
+
+## Security Notes
+
+- Change the default admin/admin credentials immediately after installation
+- The API token is displayed at the end of installation - save it securely
+- Consider restricting access to Redis on port 8001 if not needed externally
+
+## License
+
+This script is provided as-is under the same license as the Conserver vCon Server.
+
+## Contributing
+
+Feel free to submit issues or pull requests to improve this installation script.