Skip to content

Commit adfd85f

Browse files
mcgrofmcgrof
committed
workflows: add AI workflow with container-based Milvus vector database
Introduce a new AI workflow to kdevops that provides infrastructure for testing and benchmarking AI components, starting with vector database support. The design is modular and scalable to accommodate future AI components like LLMs, embeddings, training, and inference services. Architecture Overview: - Hierarchical organization separating workflows, playbooks, and roles - Consistent naming conventions across Kconfig, Make targets, and Ansible - Container-based deployment for simplified dependency management - Integration with kdevops patterns (A/B testing, baselines, CI) Directory Structure: workflows/ai/ # Workflow configuration and make targets ├── Kconfig # Configuration options ├── Makefile # Make targets └── README.md # Documentation playbooks/ai/ # Playbook organization ├── orchestrate.yml # Main orchestration ├── tests.yml # Benchmark execution └── [component]/ # Component-specific playbooks playbooks/roles/ # Ansible implementations └── milvus/ # Milvus role (shared across workflows) ├── defaults/ # Default variables ├── tasks/ # Task definitions ├── templates/ # Config templates └── files/ # Helper scripts Milvus Implementation: - Docker Compose deployment with etcd and MinIO for metadata and storage - Container-only approach eliminates complex build dependencies - Comprehensive benchmarking for insert, search, and index operations - Scalable from 100K vectors (quick test) to 1M+ vectors (full test) - Multiple index types: IVF_FLAT, HNSW with configurable parameters - Performance visualization and results collection Key Features: - Kconfig-driven configuration with CLI override support - Separated setup (make ai) from testing (make ai-tests) - Idempotent playbooks allowing repeated execution - Quick test mode for CI/demo environments - Results collection with JSON output for analysis Usage: # Configure and deploy make defconfig-ai make make bringup make ai # Install and start services # Run benchmarks make ai-tests # Execute performance tests make ai-results # Collect and analyze results # Quick CI testing (1 minute validation) AI_VECTOR_DB_MILVUS_QUICK_TEST=y make defconfig-ai make ai-tests Container Architecture: - milvus-standalone: Main vector database service - milvus-etcd: Metadata storage - milvus-minio: Object storage for vectors - Shared Docker network for inter-container communication - Persistent volumes for data retention This foundation enables systematic testing of AI infrastructure components with the same rigor applied to kernel and storage testing in kdevops. Generated-by: Claude AI Signed-off-by: Luis Chamberlain <[email protected]>
1 parent 12fa7a3 commit adfd85f

File tree

81 files changed

+11548
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+11548
-0
lines changed

‎README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ Table of Contents
1414
* [reboot-limit](#reboot-limit)
1515
* [sysbench](#sysbench)
1616
* [fio-tests](#fio-tests)
17+
* [AI workflow](#ai-workflow)
1718
* [kdevops chats](#kdevops-chats)
1819
* [kdevops on discord](#kdevops-on-discord)
1920
* [kdevops IRC](#kdevops-irc)
@@ -273,6 +274,22 @@ A/B testing capabilities, and advanced graphing and visualization support. For
273274
detailed configuration and usage information, refer to the
274275
[kdevops fio-tests documentation](docs/fio-tests.md).
275276

277+
### AI workflow
278+
279+
kdevops now supports AI/ML system benchmarking, starting with vector databases
280+
like Milvus. Similar to fstests, you can quickly set up and benchmark AI
281+
infrastructure with just a few commands:
282+
283+
```bash
284+
make defconfig-ai-milvus-docker
285+
make bringup
286+
make ai
287+
```
288+
289+
The AI workflow supports A/B testing, filesystem performance impact analysis,
290+
and comprehensive benchmarking of vector similarity search workloads. For
291+
details, see the [kdevops AI workflow documentation](docs/ai/README.md).
292+
276293
## kdevops chats
277294

278295
We use discord and IRC. Right now we have more folks on discord than on IRC.
@@ -324,6 +341,7 @@ want to just use the kernel that comes with your Linux distribution.
324341
* [kdevops NFS docs](docs/nfs.md)
325342
* [kdevops selftests docs](docs/selftests.md)
326343
* [kdevops reboot-limit docs](docs/reboot-limit.md)
344+
* [kdevops AI workflow docs](docs/ai/README.md)
327345

328346
# kdevops general documentation
329347

‎defconfigs/ai-milvus-docker‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# AI benchmarking configuration for Milvus vector database testing
2+
CONFIG_KDEVOPS_FIRST_RUN=n
3+
CONFIG_LIBVIRT=y
4+
CONFIG_LIBVIRT_URI="qemu:///system"
5+
CONFIG_LIBVIRT_HOST_PASSTHROUGH=y
6+
CONFIG_LIBVIRT_MACHINE_TYPE_DEFAULT=y
7+
CONFIG_LIBVIRT_CPU_MODEL_PASSTHROUGH=y
8+
CONFIG_LIBVIRT_VCPUS=4
9+
CONFIG_LIBVIRT_RAM=8192
10+
CONFIG_LIBVIRT_OS_VARIANT="generic"
11+
CONFIG_LIBVIRT_STORAGE_POOL_PATH_CUSTOM=n
12+
CONFIG_LIBVIRT_STORAGE_POOL_CREATE=y
13+
CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y
14+
CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE="100"
15+
16+
# Network configuration
17+
CONFIG_KDEVOPS_NETWORK_TYPE_NATUAL_BRIDGE=y
18+
19+
# Workflow configuration
20+
CONFIG_WORKFLOWS=y
21+
CONFIG_WORKFLOWS_TESTS=y
22+
CONFIG_WORKFLOWS_LINUX_TESTS=y
23+
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
24+
CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
25+
26+
# AI workflow configuration
27+
CONFIG_AI_TESTS_VECTOR_DATABASE=y
28+
CONFIG_AI_VECTOR_DB_MILVUS=y
29+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
30+
31+
# Milvus Docker configuration
32+
CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_IMAGE_2_5=y
33+
CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_IMAGE_STRING="milvusdb/milvus:v2.5.10"
34+
CONFIG_AI_VECTOR_DB_MILVUS_CONTAINER_NAME="milvus-ai-benchmark"
35+
CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CONTAINER_IMAGE_STRING="quay.io/coreos/etcd:v3.5.18"
36+
CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CONTAINER_NAME="milvus-etcd"
37+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONTAINER_IMAGE_STRING="minio/minio:RELEASE.2023-03-20T20-16-18Z"
38+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONTAINER_NAME="milvus-minio"
39+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_ACCESS_KEY="minioadmin"
40+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_SECRET_KEY="minioadmin"
41+
42+
# Docker storage configuration
43+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_DATA_PATH="/data/milvus-data"
44+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_ETCD_DATA_PATH="/data/milvus-etcd"
45+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_MINIO_DATA_PATH="/data/milvus-minio"
46+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER_NETWORK_NAME="milvus-network"
47+
48+
# Docker ports
49+
CONFIG_AI_VECTOR_DB_MILVUS_PORT=19530
50+
CONFIG_AI_VECTOR_DB_MILVUS_WEB_UI_PORT=9091
51+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_API_PORT=9000
52+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_CONSOLE_PORT=9001
53+
CONFIG_AI_VECTOR_DB_MILVUS_ETCD_CLIENT_PORT=2379
54+
CONFIG_AI_VECTOR_DB_MILVUS_ETCD_PEER_PORT=2380
55+
56+
# Docker resource limits
57+
CONFIG_AI_VECTOR_DB_MILVUS_MEMORY_LIMIT="8g"
58+
CONFIG_AI_VECTOR_DB_MILVUS_CPU_LIMIT="4.0"
59+
CONFIG_AI_VECTOR_DB_MILVUS_ETCD_MEMORY_LIMIT="1g"
60+
CONFIG_AI_VECTOR_DB_MILVUS_MINIO_MEMORY_LIMIT="2g"
61+
62+
# Milvus connection configuration
63+
CONFIG_AI_VECTOR_DB_MILVUS_COLLECTION_NAME="benchmark_collection"
64+
CONFIG_AI_VECTOR_DB_MILVUS_DIMENSION=768
65+
CONFIG_AI_VECTOR_DB_MILVUS_DATASET_SIZE=1000000
66+
CONFIG_AI_VECTOR_DB_MILVUS_BATCH_SIZE=10000
67+
CONFIG_AI_VECTOR_DB_MILVUS_NUM_QUERIES=10000
68+
69+
# Benchmark configuration
70+
CONFIG_AI_BENCHMARK_ITERATIONS=3
71+
# Vector dataset configuration
72+
CONFIG_AI_VECTOR_DB_MILVUS_DIMENSION=128
73+
74+
# Test runtime configuration
75+
CONFIG_AI_BENCHMARK_RUNTIME="180"
76+
CONFIG_AI_BENCHMARK_WARMUP_TIME="30"
77+
78+
# Query patterns for CI testing
79+
CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
80+
CONFIG_AI_BENCHMARK_QUERY_TOPK_10=y
81+
CONFIG_AI_BENCHMARK_QUERY_TOPK_100=n
82+
83+
# Batch size configuration for CI
84+
CONFIG_AI_BENCHMARK_BATCH_1=y
85+
CONFIG_AI_BENCHMARK_BATCH_10=y
86+
CONFIG_AI_BENCHMARK_BATCH_100=n
87+
88+
# Index configuration
89+
CONFIG_AI_INDEX_HNSW=y
90+
CONFIG_AI_INDEX_TYPE="HNSW"
91+
CONFIG_AI_INDEX_HNSW_M=16
92+
CONFIG_AI_INDEX_HNSW_EF_CONSTRUCTION=200
93+
CONFIG_AI_INDEX_HNSW_EF=64
94+
95+
# Results and graphing
96+
CONFIG_AI_BENCHMARK_RESULTS_DIR="/data/ai-benchmark"
97+
CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
98+
CONFIG_AI_BENCHMARK_GRAPH_FORMAT="png"
99+
CONFIG_AI_BENCHMARK_GRAPH_DPI=300
100+
CONFIG_AI_BENCHMARK_GRAPH_THEME="default"
101+
102+
# Filesystem configuration
103+
CONFIG_AI_FILESYSTEM_XFS=y
104+
CONFIG_AI_FILESYSTEM="xfs"
105+
CONFIG_AI_FSTYPE="xfs"
106+
CONFIG_AI_XFS_MKFS_OPTS="-f -s size=4096"
107+
CONFIG_AI_XFS_MOUNT_OPTS="rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota"
108+
109+
# Baseline/dev testing setup
110+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y

‎defconfigs/ai-milvus-docker-ci‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# SPDX-License-Identifier: copyleft-next-0.3.1
2+
#
3+
# AI vector database benchmarking for CI testing
4+
# Uses minimal dataset size and short runtime for quick verification
5+
6+
CONFIG_KDEVOPS_FIRST_RUN=y
7+
CONFIG_GUESTFS=y
8+
CONFIG_GUESTFS_DEBIAN=y
9+
CONFIG_GUESTFS_DEBIAN_TRIXIE=y
10+
11+
# Enable AI workflow
12+
CONFIG_WORKFLOWS_TESTS=y
13+
CONFIG_WORKFLOWS_LINUX_TESTS=y
14+
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
15+
CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
16+
CONFIG_AI_TESTS_VECTOR_DATABASE=y
17+
18+
# Docker deployment
19+
CONFIG_AI_VECTOR_DB_MILVUS=y
20+
CONFIG_AI_VECTOR_DB_MILVUS_DOCKER=y
21+
22+
# CI-optimized: Use custom small dataset
23+
CONFIG_AI_DATASET_CUSTOM=y
24+
25+
# Small vector dimensions for faster processing
26+
CONFIG_AI_VECTOR_DIM_128=y
27+
28+
# Minimal query configurations
29+
CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
30+
CONFIG_AI_BENCHMARK_BATCH_1=y
31+
32+
# Fast HNSW indexing
33+
CONFIG_AI_INDEX_HNSW=y
34+
35+
# Short runtime for CI
36+
# These will be overridden by environment variables in CI:
37+
# AI_VECTOR_DATASET_SIZE=1000
38+
# AI_BENCHMARK_RUNTIME=30
39+
40+
# Reduced resource limits for CI
41+
CONFIG_AI_VECTOR_DB_MILVUS_MEMORY_LIMIT="2g"
42+
CONFIG_AI_VECTOR_DB_MILVUS_CPU_LIMIT="2.0"
43+
44+
# Enable graphing for result verification
45+
CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
46+
47+
# XFS filesystem (fastest for AI workloads)
48+
CONFIG_AI_FILESYSTEM_XFS=y
49+
50+
# A/B testing enabled for baseline/dev comparison
51+
CONFIG_KDEVOPS_BASELINE_AND_DEV=y

‎defconfigs/ai-milvus-multifs‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# AI Multi-Filesystem Performance Testing Configuration
2+
# This configuration enables testing AI workloads across multiple filesystem
3+
# configurations including XFS (4k and 16k block sizes), ext4 (4k and 16k bigalloc),
4+
# and btrfs (default profile) to compare performance characteristics.
5+
6+
# Base virtualization setup
7+
CONFIG_LIBVIRT=y
8+
CONFIG_LIBVIRT_MACHINE_TYPE_Q35=y
9+
CONFIG_LIBVIRT_STORAGE_POOL_PATH="/opt/kdevops/libvirt"
10+
CONFIG_LIBVIRT_ENABLE_LARGEIO=y
11+
CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_NVME=y
12+
CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE="50GiB"
13+
14+
# Network configuration
15+
CONFIG_LIBVIRT_ENABLE_BRIDGED_NETWORKING=y
16+
CONFIG_LIBVIRT_NET_NAME="kdevops"
17+
18+
# Host configuration
19+
CONFIG_KDEVOPS_HOSTS_TEMPLATE="hosts.j2"
20+
CONFIG_VAGRANT_NVME_DISK_SIZE="50GiB"
21+
22+
# Base system requirements
23+
CONFIG_WORKFLOWS=y
24+
CONFIG_WORKFLOWS_TESTS=y
25+
CONFIG_WORKFLOWS_LINUX_TESTS=y
26+
CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y
27+
CONFIG_KDEVOPS_WORKFLOW_DEDICATE_AI=y
28+
29+
# AI Workflow Configuration
30+
CONFIG_KDEVOPS_WORKFLOW_ENABLE_AI=y
31+
CONFIG_AI_TESTS_VECTOR_DATABASE=y
32+
CONFIG_AI_MILVUS_DOCKER=y
33+
CONFIG_AI_VECTOR_DB_TYPE_MILVUS=y
34+
35+
# Milvus Configuration
36+
CONFIG_AI_MILVUS_HOST="localhost"
37+
CONFIG_AI_MILVUS_PORT=19530
38+
CONFIG_AI_MILVUS_DATABASE_NAME="ai_benchmark"
39+
40+
# Test Parameters (optimized for multi-fs testing)
41+
CONFIG_AI_BENCHMARK_ITERATIONS=3
42+
CONFIG_AI_DATASET_1M=y
43+
CONFIG_AI_VECTOR_DIM_128=y
44+
CONFIG_AI_BENCHMARK_RUNTIME="300"
45+
CONFIG_AI_BENCHMARK_WARMUP_TIME="30"
46+
47+
# Query patterns
48+
CONFIG_AI_BENCHMARK_QUERY_TOPK_1=y
49+
CONFIG_AI_BENCHMARK_QUERY_TOPK_10=y
50+
CONFIG_AI_BENCHMARK_QUERY_TOPK_100=n
51+
52+
# Batch sizes
53+
CONFIG_AI_BENCHMARK_BATCH_1=y
54+
CONFIG_AI_BENCHMARK_BATCH_10=y
55+
CONFIG_AI_BENCHMARK_BATCH_100=n
56+
57+
# Index configuration
58+
CONFIG_AI_INDEX_HNSW=y
59+
CONFIG_AI_INDEX_TYPE="HNSW"
60+
CONFIG_AI_INDEX_HNSW_M=16
61+
CONFIG_AI_INDEX_HNSW_EF_CONSTRUCTION=200
62+
CONFIG_AI_INDEX_HNSW_EF=64
63+
64+
# Results and visualization
65+
CONFIG_AI_BENCHMARK_RESULTS_DIR="/data/ai-benchmark"
66+
CONFIG_AI_BENCHMARK_ENABLE_GRAPHING=y
67+
CONFIG_AI_BENCHMARK_GRAPH_FORMAT="png"
68+
CONFIG_AI_BENCHMARK_GRAPH_DPI=300
69+
CONFIG_AI_BENCHMARK_GRAPH_THEME="default"
70+
71+
# Multi-filesystem testing configuration
72+
CONFIG_AI_ENABLE_MULTIFS_TESTING=y
73+
CONFIG_AI_MULTIFS_RESULTS_DIR="/data/ai-multifs-benchmark"
74+
CONFIG_AI_MULTIFS_DEVICE="/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops1"
75+
CONFIG_AI_MULTIFS_MOUNT_POINT="/mnt/ai-multifs-test"
76+
77+
# XFS configurations
78+
CONFIG_AI_MULTIFS_TEST_XFS=y
79+
CONFIG_AI_MULTIFS_XFS_4K_4KS=y
80+
CONFIG_AI_MULTIFS_XFS_16K_4KS=y
81+
82+
# ext4 configurations
83+
CONFIG_AI_MULTIFS_TEST_EXT4=y
84+
CONFIG_AI_MULTIFS_EXT4_4K=y
85+
CONFIG_AI_MULTIFS_EXT4_16K_BIGALLOC=y
86+
87+
# btrfs configurations
88+
CONFIG_AI_MULTIFS_TEST_BTRFS=y
89+
CONFIG_AI_MULTIFS_BTRFS_DEFAULT=y
90+
91+
# Standard filesystem configuration (for comparison)
92+
CONFIG_AI_FILESYSTEM_XFS=y
93+
CONFIG_AI_FILESYSTEM="xfs"
94+
CONFIG_AI_FSTYPE="xfs"
95+
CONFIG_AI_XFS_MKFS_OPTS="-f -s size=4096"
96+
CONFIG_AI_XFS_MOUNT_OPTS="rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota"
97+
98+
# Boot configuration
99+
CONFIG_BOOTLINUX=y
100+
CONFIG_BOOTLINUX_TREE_LINUS=y
101+
CONFIG_TARGET_LINUX_REF="v6.12"
102+
CONFIG_BOOTLINUX_SHALLOW_CLONE=y
103+
104+
# Memory configuration
105+
CONFIG_LIBVIRT_MEM_MB=16384

0 commit comments

Comments
 (0)