ai: add Milvus vector database benchmarking support

Add initial AI/ML workflow infrastructure starting with Milvus
vector database benchmarking. This provides a foundation for testing
AI systems with the same rigor as existing kernel testing workflows
(fstests, blktests).

Key features:
- Docker-based Milvus deployment with etcd and MinIO
  - Supports using a dedicated drive for docker /var/lib/docker/
    including custom filesystem configurations
- Python virtual environment management for benchmark dependencies
- Comprehensive benchmarking of vector operations (insert, search, delete)
- A/B testing support for baseline vs development comparisons
- Performance visualization focusing on key metrics (QPS, latency)
- Result collection and analysis infrastructure

Performance Metrics:
The benchmarks focus on two critical vector database metrics:
- QPS (Queries Per Second): Throughput measurement for search operations
- Latency: Response time percentiles (p50, p95, p99) for operations

Recall rate measurement is challenging without ground truth data - the
correct answers must be known beforehand to measure search accuracy.
Since we generate random vectors for testing, establishing meaningful
ground truth would require careful similarity calculations that would
essentially duplicate the work being tested.

Defconfigs:
- ai-milvus-docker: Standard Docker-based Milvus deployment
- ai-milvus-docker-ci: CI-optimized with minimal dataset (1000 vectors)

Workflow integration follows kdevops patterns:
  make defconfig-ai-milvus-docker
  make bringup
  make ai         # Setup infrastructure
  make ai-tests   # Run benchmarks
  make ai-results # View results

The implementation handles proper cleanup, lock file management, and
comprehensive error handling to ensure reliable benchmark execution.

Generated-by: Claude AI
Signed-off-by: Luis Chamberlain <[email protected]>

Author: mcgrof

.gitignore

@@ -32,7 +32,6 @@ scripts/workflows/fstests/lib/__pycache__/
 scripts/workflows/blktests/lib/__pycache__/
 scripts/workflows/lib/__pycache__/
 
-
 include/
 
 # You can override role specific stuff on these
@@ -48,7 +47,9 @@ playbooks/secret.yml
 playbooks/python/workflows/fstests/__pycache__/
 playbooks/python/workflows/fstests/lib/__pycache__/
 playbooks/python/workflows/fstests/gen_results_summary.pyc
+playbooks/roles/ai_run_benchmarks/files/__pycache__/
 
+workflows/ai/results/
 workflows/pynfs/results/
 
 workflows/fstests/new_expunge_files.txt

README.md

@@ -14,6 +14,7 @@ Table of Contents
       * [reboot-limit](#reboot-limit)
       * [sysbench](#sysbench)
       * [fio-tests](#fio-tests)
+      * [AI workflow](#ai-workflow)
    * [kdevops chats](#kdevops-chats)
    * [kdevops on discord](#kdevops-on-discord)
       * [kdevops IRC](#kdevops-irc)
@@ -273,6 +274,22 @@ A/B testing capabilities, and advanced graphing and visualization support. For
 detailed configuration and usage information, refer to the
 [kdevops fio-tests documentation](docs/fio-tests.md).
 
+### AI workflow
+
+kdevops now supports AI/ML system benchmarking, starting with vector databases
+like Milvus. Similar to fstests, you can quickly set up and benchmark AI
+infrastructure with just a few commands:
+
+```bash
+make defconfig-ai-milvus-docker
+make bringup
+make ai
+```
+
+The AI workflow supports A/B testing, filesystem performance impact analysis,
+and comprehensive benchmarking of vector similarity search workloads. For
+details, see the [kdevops AI workflow documentation](docs/ai/README.md).
+
 ## kdevops chats
 
 We use discord and IRC. Right now we have more folks on discord than on IRC.
@@ -324,6 +341,7 @@ want to just use the kernel that comes with your Linux distribution.
   * [kdevops NFS docs](docs/nfs.md)
   * [kdevops selftests docs](docs/selftests.md)
   * [kdevops reboot-limit docs](docs/reboot-limit.md)
+  * [kdevops AI workflow docs](docs/ai/README.md)
 
 # kdevops general documentation
 

defconfigs/ai-milvus-docker

@@ -0,0 +1,113 @@
+# AI benchmarking configuration for Milvus vector database testing
+CONFIG_KDEVOPS_FIRST_RUN=n
+CONFIG_LIBVIRT=y
+CONFIG_LIBVIRT_URI="qemu:///system"
+CONFIG_LIBVIRT_HOST_PASSTHROUGH=y
+CONFIG_LIBVIRT_MACHINE_TYPE_DEFAULT=y
+CONFIG_LIBVIRT_CPU_MODEL_PASSTHROUGH=y
+CONFIG_LIBVIRT_VCPUS=4
+CONFIG_LIBVIRT_RAM=8192
+CONFIG_LIBVIRT_OS_VARIANT="generic"
+CONFIG_LIBVIRT_STORAGE_POOL_PATH_CUSTOM=n
+CONFIG_LIBVIRT_STORAGE_POOL_CREATE=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y
+CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE="100"
+
+# Network configuration
+CONFIG_KDEVOPS_NETWORK_TYPE_NATUAL_BRIDGE=y
+
+# Workflow configuration
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
+
+# AI workflow configuration
+CONFIG_AI_TESTS_VECTOR_DATABASE=y
+CONFIG_AI_VECTOR_DB_MILVUS=y
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
+
+# Milvus Docker configuration
+CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_IMAGE_2_5=y
+CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_IMAGE_STRING="milvusdb/milvus:v2.5.10"
+CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_NAME="milvus-ai-benchmark"
+CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CONTAINER_IMAGE_STRING="quay.io/coreos/etcd:v3.5.18"
+CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CONTAINER_NAME="milvus-etcd"
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2023-03-20T20-16-18Z"
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONTAINER_NAME="milvus-minio"
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_ACCESS_KEY="minioadmin"
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_SECRET_KEY="minioadmin"
+
+# Docker storage configuration
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_DATA_PATH="/data/milvus-data"
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_ETCD_DATA_PATH="/data/milvus-etcd"
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_MINIO_DATA_PATH="/data/milvus-minio"
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_NETWORK_NAME="milvus-network"
+
+# Docker ports
+CONFIG_AI_VECTOR_DB_MILVUS_PORT=19530
+CONFIG_AI_VECTOR_DB_MILVUS_WEB_UI_PORT=9091
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_API_PORT=9000
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONSOLE_PORT=9001
+CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CLIENT_PORT=2379
+CONFIG_AI_VECTOR_DB_MILVUS_ETCD_PEER_PORT=2380
+
+# Docker resource limits
+CONFIG_AI_VECTOR_DB_MILVUS_MEMORY_LIMIT="8g"
+CONFIG_AI_VECTOR_DB_MILVUS_CPU_LIMIT="4.0"
+CONFIG_AI_VECTOR_DB_MILVUS_ETCD_MEMORY_LIMIT="1g"
+CONFIG_AI_VECTOR_DB_MILVUS_MINIO_MEMORY_LIMIT="2g"
+
+# Milvus connection configuration
+CONFIG_AI_VECTOR_DB_MILVUS_COLLECTION_NAME="benchmark_collection"
+CONFIG_AI_VECTOR_DB_MILVUS_DIMENSION=768
+CONFIG_AI_VECTOR_DB_MILVUS_DATASET_SIZE=1000000
+CONFIG_AI_VECTOR_DB_MILVUS_BATCH_SIZE=10000
+CONFIG_AI_VECTOR_DB_MILVUS_NUM_QUERIES=10000
+
+# Benchmark configuration
+CONFIG_AI_BENCHMARK_ITERATIONS=3
+# Vector dataset configuration
+CONFIG_AI_VECTOR_DB_MILVUS_DIMENSION=128
+
+# Test runtime configuration
+CONFIG_AI_BENCHMARK_RUNTIME="180"
+CONFIG_AI_BENCHMARK_WARMUP_TIME="30"
+
+# Query patterns for CI testing
+CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
+CONFIG_AI_BENCHMARK_QUERY_TOPK_10=y
+CONFIG_AI_BENCHMARK_QUERY_TOPK_100=n
+
+# Batch size configuration for CI
+CONFIG_AI_BENCHMARK_BATCH_1=y
+CONFIG_AI_BENCHMARK_BATCH_10=y
+CONFIG_AI_BENCHMARK_BATCH_100=n
+
+# Index configuration
+CONFIG_AI_INDEX_HNSW=y
+CONFIG_AI_INDEX_TYPE="HNSW"
+CONFIG_AI_INDEX_HNSW_M=16
+CONFIG_AI_INDEX_HNSW_EF_CONSTRUCTION=200
+CONFIG_AI_INDEX_HNSW_EF=64
+
+# Results and graphing
+CONFIG_AI_BENCHMARK_RESULTS_DIR="/data/ai-benchmark"
+CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
+CONFIG_AI_BENCHMARK_GRAPH_FORMAT="png"
+CONFIG_AI_BENCHMARK_GRAPH_DPI=300
+CONFIG_AI_BENCHMARK_GRAPH_THEME="default"
+
+# Filesystem configuration
+CONFIG_AI_FILESYSTEM_XFS=y
+CONFIG_AI_FILESYSTEM="xfs"
+CONFIG_AI_FSTYPE="xfs"
+CONFIG_AI_XFS_MKFS_OPTS="-f -s size=4096"
+CONFIG_AI_XFS_MOUNT_OPTS="rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota"
+
+# Baseline/dev testing setup
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y
+# Build Linux
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y

defconfigs/ai-milvus-docker-ci

@@ -0,0 +1,51 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# AI vector database benchmarking for CI testing
+# Uses minimal dataset size and short runtime for quick verification
+
+CONFIG_KDEVOPS_FIRST_RUN=y
+CONFIG_GUESTFS=y
+CONFIG_GUESTFS_DEBIAN=y
+CONFIG_GUESTFS_DEBIAN_TRIXIE=y
+
+# Enable AI workflow
+CONFIG_WORKFLOWS_TESTS=y
+CONFIG_WORKFLOWS_LINUX_TESTS=y
+CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
+CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
+CONFIG_AI_TESTS_VECTOR_DATABASE=y
+
+# Docker deployment
+CONFIG_AI_VECTOR_DB_MILVUS=y
+CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
+
+# CI-optimized: Use custom small dataset
+CONFIG_AI_DATASET_CUSTOM=y
+
+# Small vector dimensions for faster processing
+CONFIG_AI_VECTOR_DIM_128=y
+
+# Minimal query configurations
+CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
+CONFIG_AI_BENCHMARK_BATCH_1=y
+
+# Fast HNSW indexing
+CONFIG_AI_INDEX_HNSW=y
+
+# Short runtime for CI
+# These will be overridden by environment variables in CI:
+# AI_VECTOR_DATASET_SIZE=1000
+# AI_BENCHMARK_RUNTIME=30
+
+# Reduced resource limits for CI
+CONFIG_AI_VECTOR_DB_MILVUS_MEMORY_LIMIT="2g"
+CONFIG_AI_VECTOR_DB_MILVUS_CPU_LIMIT="2.0"
+
+# Enable graphing for result verification
+CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
+
+# XFS filesystem (fastest for AI workloads)
+CONFIG_AI_FILESYSTEM_XFS=y
+
+# A/B testing enabled for baseline/dev comparison
+CONFIG_KDEVOPS_BASELINE_AND_DEV=y

docs/ai/README.md

@@ -0,0 +1,108 @@
+# AI Workflow Documentation
+
+The kdevops AI workflow provides infrastructure for benchmarking and testing AI/ML systems, with initial support for vector databases.
+
+## Quick Start
+
+Just like other kdevops workflows (fstests, blktests), the AI workflow follows the same pattern:
+
+```bash
+make defconfig-ai-milvus-docker # Configure for AI vector database testing
+make bringup # Bring up the test environment
+make ai # Run the AI benchmarks
+make ai-baseline # Establish baseline results
+make ai-results # View results
+```
+
+## Supported Components
+
+### Vector Databases
+- [Milvus](vector-databases/milvus.md) - High-performance vector database for AI applications
+
+### Future Components (Planned)
+- Language Models (LLMs)
+- Embedding Services
+- Training Infrastructure
+- Inference Servers
+
+## Configuration Options
+
+The AI workflow can be configured through `make menuconfig`:
+
+1. **Vector Database Selection**
+   - Milvus (Docker or Native deployment)
+   - Future: Weaviate, Qdrant, Pinecone
+
+2. **Dataset Configuration**
+   - Dataset size (number of vectors)
+   - Vector dimensions
+   - Batch sizes
+
+3. **Benchmark Parameters**
+   - Query patterns
+   - Concurrency levels
+   - Runtime duration
+
+4. **Filesystem Testing**
+   - Test on different filesystems (XFS, ext4, btrfs)
+   - Compare performance across storage configurations
+
+## Pre-built Configurations
+
+Quick configurations for common use cases:
+
+- `defconfig-ai-milvus-docker` - Docker-based Milvus deployment
+- `defconfig-ai-milvus-docker-ci` - CI-optimized with minimal dataset
+- `defconfig-ai-milvus-native` - Native Milvus installation from source
+- `defconfig-ai-milvus-multifs` - Multi-filesystem performance comparison
+
+## A/B Testing Support
+
+Like other kdevops workflows, AI supports baseline/dev comparisons:
+
+```bash
+# Configure with A/B testing
+make menuconfig  # Enable CONFIG_KDEVOPS_BASELINE_AND_DEV
+make ai-baseline # Run on baseline
+make ai-dev # Run on dev
+make ai-results # Compare results
+```
+
+## Results and Analysis
+
+The AI workflow generates comprehensive performance metrics:
+
+- Throughput (operations/second)
+- Latency percentiles (p50, p95, p99)
+- Resource utilization
+- Performance graphs and trends
+
+Results are stored in the configured results directory (default: `/data/ai-results/`).
+
+## Integration with CI/CD
+
+The workflow includes CI-optimized configurations that use:
+- Minimal datasets for quick validation
+- `/dev/null` storage for I/O testing without disk requirements
+- Environment variable overrides for runtime configuration
+
+Example CI usage:
+```bash
+AI_VECTOR_DATASET_SIZE=1000 AI_BENCHMARK_RUNTIME=30 make defconfig-ai-milvus-docker-ci
+make bringup
+make ai
+```
+
+## Workflow Architecture
+
+The AI workflow follows kdevops patterns:
+
+1. **Configuration** - Kconfig-based configuration system
+2. **Provisioning** - Ansible-based infrastructure setup
+3. **Execution** - Standardized test execution
+4. **Collection** - Automated result collection and analysis
+5. **Reporting** - Performance visualization and comparison
+
+For detailed usage of specific components, see:
+- [Vector Databases Overview](vector-databases/README.md)
+- [Milvus Usage Guide](vector-databases/milvus.md)