A cloud-native, serverless vector analytics engine with separation of compute and storage. Built with Go + Rust for maximum performance.
VexLake follows the "Sandwich Architecture" - Go handles the network facade, Rust handles all heavy computation:
┌─────────────────────────────────────────────────────────────────┐
│ Go Layer (Network Facade) │
│ • RESP protocol (Redis-compatible) │
│ • Client connection management │
│ • Result formatting (Arrow → RESP) │
└───────────────────────────┬─────────────────────────────────────┘
│ CGO / FFI (Arrow C Data Interface)
▼
┌─────────────────────────────────────────────────────────────────┐
│ Rust Core (libvector_engine.so) │
│ • SIMD-accelerated vector search (AVX-512/NEON) │
│ • HNSW/IVF index management │
│ • Parquet read/write via DataFusion │
│ • S3 I/O via OpenDAL │
└───────────────────────────┬─────────────────────────────────────┘
│ S3 API
▼
┌─────────────────────────────────────────────────────────────────┐
│ SeaweedFS Storage │
│ • Parquet data files │
│ • Binary index files │
│ • Version metadata │
└─────────────────────────────────────────────────────────────────┘
- Redis Protocol Compatible: Connect with any Redis client (redis-cli, redis-py, Jedis)
- SIMD-Accelerated Search: 2-10x faster vector computation using AVX-512/NEON
- Zero-Copy Interface: Arrow C Data Interface between Go and Rust
- Cloud-Native Storage: SeaweedFS backend with S3 API compatibility
- Separation of Compute and Storage: Scale compute and storage independently
- MVCC Transactions: Snapshot isolation via versioned metadata
# Build everything
make build
# Run the server
make run
# Run tests
make test
# Run all CI checks locally
make verify# Build the image
make docker-build
# Run the container
make docker-runredis-cli -p 6379
# Test connection
127.0.0.1:6379> PING
PONG
# Store a vector
127.0.0.1:6379> VSET vec:1 "[0.1, 0.2, 0.3, 0.4]"
OK
# Retrieve a vector
127.0.0.1:6379> VGET vec:1
"[0.100000, 0.200000, 0.300000, 0.400000]"
# Search for similar vectors
127.0.0.1:6379> VSEARCH "[0.1, 0.2, 0.3, 0.4]" 5
1) "vec:1"
2) "vec:42"
3) "vec:17"| Command | Description |
|---|---|
PING [message] |
Test connection |
ECHO message |
Echo back a message |
STATS / INFO |
Get server statistics |
QUIT |
Close connection |
| Command | Description |
|---|---|
VSET key "[...]" |
Store a vector |
VGET key |
Retrieve a vector |
VSEARCH "[...]" K |
Find top K similar vectors |
VDEL key |
Delete a vector |
CLEAR |
Remove all vectors |
vexlake/
├── cmd/
│ ├── vexlake-server/ # Go RESP server
│ └── vexlake-bench/ # Go benchmark tool
├── crates/
│ ├── vexlake-core/ # Rust core library
│ └── vexlake-bench/ # Rust benchmark tool
├── docs/
│ ├── 01_vexlake_design.md # Architecture design
│ ├── 02_seaweedfs_integration.md # Storage integration
│ └── 03_development_plan.md # Development roadmap
├── .github/workflows/ # CI/CD
├── Cargo.toml # Rust workspace
├── go.mod # Go module
└── Makefile # Build commands
make help # Show all available commands
make build # Build Rust library and Go binaries
make test # Run all tests (Rust + Go)
make fmt # Format all code
make clippy # Run Rust linter
make lint # Run all linters
make verify # Run all CI checks locally
make doc # Generate Rust documentation
make clean # Clean build artifacts# Rust vector operation benchmarks
make run-rust-bench
# Go client benchmarks
./bin/vexlake-bench -mode=insert -n=100000 -concurrency=50
./bin/vexlake-bench -mode=search -n=50000 -concurrency=50| Metric | Value |
|---|---|
| Insert Throughput | >50,000 ops/sec |
| Search Latency (P99) | <10ms |
| TopK Computation | 2-10x faster than Go (SIMD) |
| Memory Transfer | Zero-copy (Arrow C Interface) |
- Architecture Design - Detailed design specification
- SeaweedFS Integration - Storage layer details
- Development Plan - Roadmap and milestones
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
See CONTRIBUTING.md for contribution guidelines.
- DataFusion - Rust SQL execution engine
- OpenDAL - Rust storage abstraction
- SeaweedFS - Distributed file system
- Arrow - Cross-language data format
- tidwall/redcon - Go RESP server framework