NeuraFind

A next-generation distributed search and vector database built for high performance, scalability, and developer experience.

Overview

NeuraFind is a distributed, high-performance search and vector database that improves upon Elasticsearch with better performance, cost-efficiency, and developer experience. It supports hybrid search combining full-text, vector, and structured data with a custom LSM-tree storage engine.

Architecture

• Coordinator: REST/gRPC API, query routing, and result aggregation
• Data Node: Shard management and local query execution
• Storage Engine: Custom LSM-tree with vector indexing (Rust)
• Metadata Store: etcd for cluster state and coordination

Tech Stack

• Go: Coordinator and DataNode services
• Rust: Storage engine (LSM-tree, vector search)
• etcd: Cluster metadata and coordination
• gRPC: Inter-service communication
• Protocol Buffers: API contracts and code generation

Quick Start

Prerequisites

• Go 1.21+
• Rust 1.70+
• Protocol Buffers compiler
• etcd (for clustering)

Development Setup

# Clone the repository
git clone <repository-url>
cd SearchDb

# Install dependencies
make setup

# Start local development cluster
make dev-start

# Run tests
make test

Project Structure

SearchDb/
├── coordinator/          # Go - REST/gRPC API, query routing
├── datanode/            # Go - shard management, local query
├── storage-engine/      # Rust - LSM-tree, compaction, vector index
├── proto/              # Protocol buffers definitions
├── metadata/           # Go - etcd integration, cluster state
├── common/             # Go - shared types, schema DSL
├── docs/               # Documentation
├── scripts/            # Build, test, deployment scripts
├── docker/             # Dockerfiles for all components
└── k8s/                # Helm charts, K8s manifests

API Examples

Index a Document

curl -X POST "http://localhost:8080/index/books/_doc" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Building Scalable Systems",
    "description": "Guide to distributed architecture...",
    "category": "tech",
    "price": 29.99,
    "embedding": [0.12, 0.34, ...]
  }'

Search Documents

curl -X POST "http://localhost:8080/index/books/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "hybrid": {
        "text": { "match": { "title": "distributed systems" } },
        "vector": { "field": "embedding", "query_vector": [0.1, 0.2, ...] },
        "filter": { "term": { "category": "tech" } }
      }
    },
    "size": 10
  }'

Development

See docs/implementation-checklist.md for detailed implementation progress and roadmap.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

License

[License information to be added]