Skip to content

Latest commit

 

History

History
1661 lines (1373 loc) · 41.9 KB

NPM_PACKAGE_ARCHITECTURE.md

File metadata and controls

1661 lines (1373 loc) · 41.9 KB

Ruvector NPM Package Architecture Design

Version: 1.0.0 Status: Design Document Author: System Architecture Designer Date: 2025-11-20

Table of Contents

  1. Executive Summary
  2. Architecture Overview
  3. Package Structure
  4. Platform Support Matrix
  5. Directory Structure
  6. Package Specifications
  7. Platform Detection Logic
  8. Build & Publish Workflow
  9. TypeScript Definitions
  10. API Consistency
  11. Implementation Roadmap
  12. ADRs (Architecture Decision Records)

Executive Summary

This document outlines a modular npm package architecture for Ruvector, a high-performance Rust-native vector database. The architecture provides:

  • Platform-specific optimizations via NAPI-RS native bindings
  • Universal fallback via WebAssembly for unsupported platforms
  • Modular design allowing users to install only what they need
  • Automatic platform detection with zero-config setup
  • Consistent API across NAPI-RS and WASM implementations

Key Packages

Package Purpose Size Platforms
ruvector Main package with auto-detection ~50KB All
@ruvector/core Native bindings (platform-specific) ~5-8MB linux-x64, linux-arm64, darwin-x64, darwin-arm64, win32-x64
@ruvector/wasm WebAssembly fallback ~1.5MB Universal
@ruvector/cli CLI tools ~8MB linux-x64, linux-arm64, darwin-x64, darwin-arm64, win32-x64

Architecture Overview

Design Principles

  1. Zero-Configuration: Users install ruvector and it just works
  2. Performance-First: Native bindings preferred, WASM as fallback
  3. Modularity: Users can opt into specific packages
  4. Consistency: Same API across native and WASM
  5. Type-Safety: Full TypeScript support
  6. Bundle Size: Main package stays minimal (~50KB)

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                         ruvector (Main)                      │
│  - Platform detection                                        │
│  - Auto-loader                                               │
│  - TypeScript definitions                                    │
└───────────────┬─────────────────────────────────────────────┘
                │
                ├─────────────────┬──────────────────┐
                │                 │                  │
        ┌───────▼──────┐  ┌──────▼──────┐  ┌───────▼──────┐
        │ @ruvector/   │  │ @ruvector/  │  │ @ruvector/   │
        │    core      │  │    wasm     │  │     cli      │
        │              │  │             │  │              │
        │ NAPI-RS      │  │ WebAssembly │  │ CLI Tools    │
        │ Native       │  │ Universal   │  │ Standalone   │
        └──────────────┘  └─────────────┘  └──────────────┘
                │                 │
                │                 │
        ┌───────▼─────────────────▼────────┐
        │     ruvector-core (Rust)          │
        │  - HNSW indexing                  │
        │  - SIMD optimizations             │
        │  - Storage engine                 │
        └───────────────────────────────────┘

Package Structure

1. ruvector (Main Package)

Purpose: Zero-config entry point with automatic platform detection and loading.

Features:

  • Detects platform and loads appropriate native module
  • Falls back to WASM if native binding unavailable
  • Re-exports consistent API
  • Minimal bundle size (~50KB)
  • TypeScript definitions included

Dependencies:

{
  "optionalDependencies": {
    "@ruvector/core-linux-x64": "^0.1.1",
    "@ruvector/core-linux-arm64": "^0.1.1",
    "@ruvector/core-darwin-x64": "^0.1.1",
    "@ruvector/core-darwin-arm64": "^0.1.1",
    "@ruvector/core-win32-x64": "^0.1.1"
  },
  "dependencies": {
    "@ruvector/wasm": "^0.1.1"
  }
}

2. @ruvector/core (Native Bindings)

Purpose: High-performance NAPI-RS native bindings with platform-specific builds.

Platform Packages:

  • @ruvector/core-linux-x64
  • @ruvector/core-linux-arm64
  • @ruvector/core-darwin-x64
  • @ruvector/core-darwin-arm64
  • @ruvector/core-win32-x64

Features:

  • Full SIMD optimizations
  • Native file I/O
  • Maximum performance
  • ~5-8MB per platform

3. @ruvector/wasm (WebAssembly Fallback)

Purpose: Universal WebAssembly implementation for unsupported platforms and browsers.

Features:

  • IndexedDB persistence
  • Web Worker support
  • SIMD WASM where available
  • ~1.5MB gzipped
  • Browser and Node.js compatible

4. @ruvector/cli (CLI Tools)

Purpose: Command-line interface for database management and operations.

Features:

  • Database creation and management
  • Import/export utilities
  • Benchmarking tools
  • MCP server support
  • Standalone binary

Platform Support Matrix

Platform Architecture Native Support WASM Fallback Package Name
Linux x64 ✅ ✅ @ruvector/core-linux-x64
Linux ARM64 ✅ ✅ @ruvector/core-linux-arm64
macOS x64 (Intel) ✅ ✅ @ruvector/core-darwin-x64
macOS ARM64 (Apple Silicon) ✅ ✅ @ruvector/core-darwin-arm64
Windows x64 ✅ ✅ @ruvector/core-win32-x64
Windows ARM64 ❌ ✅ WASM only
FreeBSD x64 ❌ ✅ WASM only
Browser Any ❌ ✅ @ruvector/wasm

Performance Characteristics:

  • Native (NAPI-RS): 100% performance baseline
  • WASM with SIMD: ~70-80% of native performance
  • WASM without SIMD: ~40-50% of native performance

Directory Structure

/workspaces/ruvector/
├── npm/                                  # NPM package root
│   ├── ruvector/                         # Main package
│   │   ├── package.json
│   │   ├── index.js                      # Platform detection & loader
│   │   ├── index.d.ts                    # TypeScript definitions
│   │   ├── lib/
│   │   │   ├── loader.js                 # Dynamic loader
│   │   │   ├── platform.js               # Platform detection
│   │   │   └── error-handler.js          # Error handling
│   │   ├── README.md
│   │   ├── LICENSE
│   │   └── CHANGELOG.md
│   │
│   ├── core/                             # Native bindings base
│   │   ├── package.json                  # Meta package
│   │   ├── index.js                      # Exports common interface
│   │   ├── index.d.ts                    # TypeScript definitions
│   │   └── README.md
│   │
│   ├── core-linux-x64/                   # Platform-specific packages
│   │   ├── package.json
│   │   ├── ruvector.linux-x64-gnu.node   # Native binary
│   │   └── README.md
│   │
│   ├── core-linux-arm64/
│   │   ├── package.json
│   │   ├── ruvector.linux-arm64-gnu.node
│   │   └── README.md
│   │
│   ├── core-darwin-x64/
│   │   ├── package.json
│   │   ├── ruvector.darwin-x64.node
│   │   └── README.md
│   │
│   ├── core-darwin-arm64/
│   │   ├── package.json
│   │   ├── ruvector.darwin-arm64.node
│   │   └── README.md
│   │
│   ├── core-win32-x64/
│   │   ├── package.json
│   │   ├── ruvector.win32-x64-msvc.node
│   │   └── README.md
│   │
│   ├── wasm/                             # WebAssembly package
│   │   ├── package.json
│   │   ├── index.js                      # WASM loader
│   │   ├── index.d.ts                    # TypeScript definitions
│   │   ├── pkg/                          # wasm-pack output
│   │   │   ├── ruvector_wasm.js
│   │   │   ├── ruvector_wasm.d.ts
│   │   │   ├── ruvector_wasm_bg.wasm
│   │   │   └── ruvector_wasm_bg.wasm.d.ts
│   │   ├── pkg-simd/                     # SIMD build
│   │   │   └── ...
│   │   ├── src/
│   │   │   ├── worker.js                 # Web Worker support
│   │   │   ├── worker-pool.js            # Worker pool
│   │   │   └── indexeddb.js              # IndexedDB persistence
│   │   ├── README.md
│   │   └── LICENSE
│   │
│   └── cli/                              # CLI package
│       ├── package.json
│       ├── bin/
│       │   ├── ruvector                  # Binary entry
│       │   └── ruvector-mcp              # MCP server entry
│       ├── lib/
│       │   ├── index.js
│       │   └── commands/
│       ├── README.md
│       └── LICENSE
│
├── crates/                               # Rust source (existing)
│   ├── ruvector-core/
│   ├── ruvector-node/                    # Builds to npm/core-*/
│   ├── ruvector-wasm/                    # Builds to npm/wasm/
│   └── ruvector-cli/                     # Builds to npm/cli/
│
└── scripts/                              # Build scripts
    ├── build-npm-packages.sh             # Build all packages
    ├── build-native.sh                   # Build native bindings
    ├── build-wasm.sh                     # Build WASM
    ├── publish-npm-packages.sh           # Publish workflow
    └── test-npm-packages.sh              # Test packages

Package Specifications

1. ruvector (Main Package)

package.json

{
  "name": "ruvector",
  "version": "0.1.1",
  "description": "High-performance Rust-native vector database with automatic platform detection",
  "main": "index.js",
  "types": "index.d.ts",
  "type": "module",
  "keywords": [
    "vector",
    "database",
    "embeddings",
    "similarity-search",
    "hnsw",
    "rust",
    "napi",
    "wasm",
    "semantic-search",
    "machine-learning",
    "rag",
    "simd"
  ],
  "author": "Ruvector Team",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/ruvnet/ruvector.git",
    "directory": "npm/ruvector"
  },
  "homepage": "https://github.com/ruvnet/ruvector#readme",
  "bugs": {
    "url": "https://github.com/ruvnet/ruvector/issues"
  },
  "engines": {
    "node": ">=18.0.0"
  },
  "files": [
    "index.js",
    "index.d.ts",
    "lib/",
    "README.md",
    "LICENSE",
    "CHANGELOG.md"
  ],
  "optionalDependencies": {
    "@ruvector/core-linux-x64": "0.1.1",
    "@ruvector/core-linux-arm64": "0.1.1",
    "@ruvector/core-darwin-x64": "0.1.1",
    "@ruvector/core-darwin-arm64": "0.1.1",
    "@ruvector/core-win32-x64": "0.1.1"
  },
  "dependencies": {
    "@ruvector/wasm": "0.1.1"
  },
  "scripts": {
    "test": "node --test",
    "postinstall": "node lib/postinstall.js"
  }
}

index.js (Loader)

/**
 * Ruvector - High-performance vector database
 * Automatically detects platform and loads native or WASM implementation
 */

import { createRequire } from 'node:module';
import { platform, arch } from 'node:os';
import { loadNative, loadWasm } from './lib/loader.js';
import { getPlatformIdentifier } from './lib/platform.js';

let binding = null;
let usingWasm = false;

/**
 * Initialize and load appropriate implementation
 */
async function initialize() {
  if (binding) return binding;

  const platformId = getPlatformIdentifier();

  try {
    // Try to load native binding first
    binding = await loadNative(platformId);
    console.debug(`[ruvector] Loaded native binding for ${platformId}`);
    return binding;
  } catch (nativeError) {
    console.debug(`[ruvector] Native binding not available: ${nativeError.message}`);

    try {
      // Fall back to WASM
      binding = await loadWasm();
      usingWasm = true;
      console.debug('[ruvector] Using WebAssembly fallback');
      return binding;
    } catch (wasmError) {
      throw new Error(
        `Failed to load ruvector:\n` +
        `Native: ${nativeError.message}\n` +
        `WASM: ${wasmError.message}`
      );
    }
  }
}

/**
 * Get implementation info
 */
export function getImplementation() {
  return {
    type: usingWasm ? 'wasm' : 'native',
    platform: getPlatformIdentifier(),
    initialized: binding !== null
  };
}

// Pre-initialize on import (for synchronous usage)
const initPromise = initialize();

// Export promise for async initialization
export { initPromise };

// Re-export all APIs
export * from './lib/loader.js';

// Default export for CJS compatibility
export default {
  initialize,
  getImplementation
};

lib/platform.js

/**
 * Platform detection utilities
 */

import { platform as osPlatform, arch as osArch } from 'node:os';

/**
 * Platform mapping for npm package names
 */
const PLATFORM_MAP = {
  'linux-x64': 'linux-x64',
  'linux-arm64': 'linux-arm64',
  'darwin-x64': 'darwin-x64',
  'darwin-arm64': 'darwin-arm64',
  'win32-x64': 'win32-x64'
};

/**
 * Architecture normalization
 */
function normalizeArch(arch) {
  switch (arch) {
    case 'x64':
    case 'x86_64':
    case 'amd64':
      return 'x64';
    case 'arm64':
    case 'aarch64':
      return 'arm64';
    default:
      return arch;
  }
}

/**
 * Get platform identifier for package lookup
 */
export function getPlatformIdentifier() {
  const platform = osPlatform();
  const arch = normalizeArch(osArch());
  const key = `${platform}-${arch}`;

  return PLATFORM_MAP[key] || null;
}

/**
 * Get package name for current platform
 */
export function getPlatformPackageName() {
  const platformId = getPlatformIdentifier();
  if (!platformId) return null;
  return `@ruvector/core-${platformId}`;
}

/**
 * Check if platform is supported natively
 */
export function isPlatformSupported() {
  return getPlatformIdentifier() !== null;
}

/**
 * Get platform info
 */
export function getPlatformInfo() {
  return {
    platform: osPlatform(),
    arch: osArch(),
    normalizedArch: normalizeArch(osArch()),
    identifier: getPlatformIdentifier(),
    supported: isPlatformSupported()
  };
}

lib/loader.js

/**
 * Dynamic loader for native and WASM implementations
 */

import { createRequire } from 'node:module';
import { getPlatformPackageName } from './platform.js';

const require = createRequire(import.meta.url);

/**
 * Load native binding for current platform
 */
export async function loadNative(platformId) {
  if (!platformId) {
    throw new Error('Platform not supported for native bindings');
  }

  const packageName = `@ruvector/core-${platformId}`;

  try {
    const nativeBinding = require(packageName);
    return nativeBinding;
  } catch (error) {
    throw new Error(
      `Failed to load native binding '${packageName}': ${error.message}. ` +
      `Try reinstalling or use WASM fallback.`
    );
  }
}

/**
 * Load WASM implementation
 */
export async function loadWasm() {
  try {
    const wasm = await import('@ruvector/wasm');
    await wasm.default(); // Initialize WASM
    return wasm;
  } catch (error) {
    throw new Error(`Failed to load WASM implementation: ${error.message}`);
  }
}

/**
 * Try loading in order: native -> WASM
 */
export async function loadAny() {
  const platformId = getPlatformPackageName();

  try {
    return await loadNative(platformId);
  } catch {
    return await loadWasm();
  }
}

2. @ruvector/core-* (Platform-Specific Packages)

package.json Template

{
  "name": "@ruvector/core-linux-x64",
  "version": "0.1.1",
  "description": "Ruvector native binding for Linux x64",
  "main": "ruvector.linux-x64-gnu.node",
  "os": ["linux"],
  "cpu": ["x64"],
  "keywords": [
    "ruvector",
    "native",
    "napi",
    "vector-database"
  ],
  "author": "Ruvector Team",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/ruvnet/ruvector.git",
    "directory": "npm/core-linux-x64"
  },
  "engines": {
    "node": ">=18.0.0"
  },
  "files": [
    "ruvector.linux-x64-gnu.node",
    "README.md"
  ]
}

3. @ruvector/wasm

package.json

{
  "name": "@ruvector/wasm",
  "version": "0.1.1",
  "description": "Ruvector WebAssembly implementation for universal compatibility",
  "main": "index.js",
  "types": "index.d.ts",
  "type": "module",
  "keywords": [
    "ruvector",
    "wasm",
    "webassembly",
    "vector-database",
    "browser"
  ],
  "author": "Ruvector Team",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/ruvnet/ruvector.git",
    "directory": "npm/wasm"
  },
  "engines": {
    "node": ">=18.0.0"
  },
  "files": [
    "index.js",
    "index.d.ts",
    "pkg/",
    "pkg-simd/",
    "src/worker.js",
    "src/worker-pool.js",
    "src/indexeddb.js",
    "README.md",
    "LICENSE"
  ],
  "exports": {
    ".": {
      "import": "./index.js",
      "types": "./index.d.ts"
    },
    "./worker": "./src/worker.js",
    "./worker-pool": "./src/worker-pool.js",
    "./indexeddb": "./src/indexeddb.js"
  },
  "scripts": {
    "test": "node --test"
  }
}

index.js (WASM Loader)

/**
 * Ruvector WASM implementation
 * Universal WebAssembly bindings
 */

let wasm = null;
let wasmSimd = null;

/**
 * Detect SIMD support
 */
function detectSimd() {
  try {
    return WebAssembly.validate(new Uint8Array([
      0x00, 0x61, 0x73, 0x6d, 0x01, 0x00, 0x00, 0x00,
      0x01, 0x05, 0x01, 0x60, 0x00, 0x01, 0x7b,
      0x03, 0x02, 0x01, 0x00,
      0x0a, 0x0a, 0x01, 0x08, 0x00, 0x41, 0x00, 0xfd,
      0x0f, 0x0b
    ]));
  } catch {
    return false;
  }
}

/**
 * Initialize WASM module
 */
async function initWasm() {
  if (wasm) return wasm;

  const hasSimd = detectSimd();

  try {
    if (hasSimd) {
      // Try SIMD build first
      wasmSimd = await import('./pkg-simd/ruvector_wasm.js');
      await wasmSimd.default();
      wasm = wasmSimd;
      console.debug('[ruvector-wasm] Loaded SIMD build');
    }
  } catch (error) {
    console.debug('[ruvector-wasm] SIMD build not available:', error.message);
  }

  if (!wasm) {
    // Fall back to regular build
    const wasmRegular = await import('./pkg/ruvector_wasm.js');
    await wasmRegular.default();
    wasm = wasmRegular;
    console.debug('[ruvector-wasm] Loaded regular build');
  }

  return wasm;
}

/**
 * Export default initializer
 */
export default initWasm;

/**
 * Get WASM info
 */
export function getWasmInfo() {
  return {
    initialized: wasm !== null,
    simdEnabled: wasmSimd !== null,
    simdSupported: detectSimd()
  };
}

// Re-export APIs (lazy initialization)
export const VectorDatabase = (...args) => initWasm().then(w => new w.VectorDatabase(...args));
export const Collection = (...args) => initWasm().then(w => new w.Collection(...args));
// ... other exports

4. @ruvector/cli

package.json

{
  "name": "@ruvector/cli",
  "version": "0.1.1",
  "description": "CLI tools for Ruvector vector database",
  "type": "module",
  "bin": {
    "ruvector": "bin/ruvector",
    "ruvector-mcp": "bin/ruvector-mcp"
  },
  "keywords": [
    "ruvector",
    "cli",
    "vector-database",
    "mcp"
  ],
  "author": "Ruvector Team",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/ruvnet/ruvector.git",
    "directory": "npm/cli"
  },
  "engines": {
    "node": ">=18.0.0"
  },
  "files": [
    "bin/",
    "lib/",
    "README.md",
    "LICENSE"
  ],
  "dependencies": {
    "ruvector": "^0.1.1"
  },
  "optionalDependencies": {
    "@ruvector/core-linux-x64": "0.1.1",
    "@ruvector/core-linux-arm64": "0.1.1",
    "@ruvector/core-darwin-x64": "0.1.1",
    "@ruvector/core-darwin-arm64": "0.1.1",
    "@ruvector/core-win32-x64": "0.1.1"
  }
}

Platform Detection Logic

Detection Flow

┌─────────────────────────┐
│  Application Import     │
│  import ruvector from   │
│  'ruvector'             │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│  Detect Platform        │
│  - OS: linux/darwin/win │
│  - Arch: x64/arm64      │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│  Try Load Native        │
│  @ruvector/core-{plat}  │
└───────────┬─────────────┘
            │
        ┌───┴───┐
        │       │
    Success   Failure
        │       │
        │       ▼
        │  ┌─────────────────────────┐
        │  │  Try Load WASM          │
        │  │  @ruvector/wasm         │
        │  └───────────┬─────────────┘
        │              │
        │          ┌───┴───┐
        │          │       │
        │      Success   Failure
        │          │       │
        ▼          ▼       ▼
    ┌────────────────────────────┐
    │  Return Binding            │
    │  (Native or WASM)          │
    └────────────────────────────┘
                │
                ▼
    ┌────────────────────────────┐
    │  Initialize Error          │
    │  (No compatible runtime)   │
    └────────────────────────────┘

Implementation Strategy

  1. Synchronous Detection: Platform detection at module load time
  2. Asynchronous Loading: Dynamic import for actual implementation
  3. Graceful Degradation: Native → WASM → Error
  4. Cache Results: Store loaded binding to avoid re-detection
  5. Debug Logging: Console debug messages for troubleshooting

Error Handling

try {
  const db = await ruvector.createDatabase('mydb');
} catch (error) {
  if (error.code === 'PLATFORM_UNSUPPORTED') {
    console.error('Your platform is not supported. Install @ruvector/wasm manually.');
  } else if (error.code === 'NATIVE_LOAD_FAILED') {
    console.error('Native binding failed to load. Falling back to WASM.');
  } else {
    throw error;
  }
}

Build & Publish Workflow

Build Process

# scripts/build-npm-packages.sh

#!/bin/bash
set -e

echo "Building Ruvector NPM packages..."

# 1. Build Rust crates
echo "Building Rust crates..."
cargo build --release --workspace

# 2. Build native bindings for all platforms
echo "Building native bindings..."
./scripts/build-native.sh

# 3. Build WASM packages
echo "Building WASM packages..."
./scripts/build-wasm.sh

# 4. Create npm package structure
echo "Creating npm packages..."
./scripts/create-npm-packages.sh

# 5. Copy TypeScript definitions
echo "Copying TypeScript definitions..."
./scripts/copy-typedefs.sh

# 6. Run tests
echo "Testing packages..."
./scripts/test-npm-packages.sh

echo "Build complete!"

scripts/build-native.sh

#!/bin/bash
set -e

echo "Building native bindings for all platforms..."

# Define platforms
PLATFORMS=(
  "x86_64-unknown-linux-gnu:linux-x64"
  "aarch64-unknown-linux-gnu:linux-arm64"
  "x86_64-apple-darwin:darwin-x64"
  "aarch64-apple-darwin:darwin-arm64"
  "x86_64-pc-windows-msvc:win32-x64"
)

cd crates/ruvector-node

for platform_pair in "${PLATFORMS[@]}"; do
  IFS=':' read -r rust_target npm_platform <<< "$platform_pair"

  echo "Building for $rust_target ($npm_platform)..."

  # Build with cargo
  cargo build --release --target "$rust_target"

  # Copy to npm directory
  mkdir -p "../../npm/core-$npm_platform"

  # Determine file extension
  if [[ $npm_platform == win32-* ]]; then
    ext=".dll"
    node_ext=".node"
  else
    ext=".so"
    if [[ $npm_platform == darwin-* ]]; then
      ext=".dylib"
    fi
    node_ext=".node"
  fi

  # Copy and rename
  cp "../../target/$rust_target/release/libruvector_node$ext" \
     "../../npm/core-$npm_platform/ruvector.$npm_platform$node_ext"

  # Create package.json
  ./scripts/create-platform-package.sh "$npm_platform"
done

echo "Native bindings built successfully!"

scripts/build-wasm.sh

#!/bin/bash
set -e

echo "Building WASM packages..."

cd crates/ruvector-wasm

# Build regular WASM
echo "Building regular WASM..."
wasm-pack build --target web --out-dir ../../npm/wasm/pkg --release

# Build SIMD WASM
echo "Building SIMD WASM..."
wasm-pack build --target web --out-dir ../../npm/wasm/pkg-simd --release -- --features simd

# Optimize WASM files
echo "Optimizing WASM..."
wasm-opt -Oz ../../npm/wasm/pkg/ruvector_wasm_bg.wasm -o ../../npm/wasm/pkg/ruvector_wasm_bg.wasm
wasm-opt -Oz ../../npm/wasm/pkg-simd/ruvector_wasm_bg.wasm -o ../../npm/wasm/pkg-simd/ruvector_wasm_bg.wasm

# Copy worker files
echo "Copying worker support files..."
cp src/worker.js ../../npm/wasm/src/
cp src/worker-pool.js ../../npm/wasm/src/
cp src/indexeddb.js ../../npm/wasm/src/

# Create package.json
cd ../../npm/wasm
cat > package.json << 'EOF'
{
  "name": "@ruvector/wasm",
  "version": "0.1.1",
  ...
}
EOF

echo "WASM packages built successfully!"

Publish Workflow

# scripts/publish-npm-packages.sh

#!/bin/bash
set -e

VERSION=${1:-"0.1.1"}

echo "Publishing Ruvector packages version $VERSION..."

# 1. Publish platform-specific packages first
PLATFORMS=(
  "linux-x64"
  "linux-arm64"
  "darwin-x64"
  "darwin-arm64"
  "win32-x64"
)

for platform in "${PLATFORMS[@]}"; do
  echo "Publishing @ruvector/core-$platform..."
  cd "npm/core-$platform"
  npm publish --access public
  cd ../..
done

# 2. Publish WASM package
echo "Publishing @ruvector/wasm..."
cd npm/wasm
npm publish --access public
cd ../..

# 3. Publish CLI package
echo "Publishing @ruvector/cli..."
cd npm/cli
npm publish --access public
cd ../..

# 4. Publish main package last
echo "Publishing ruvector..."
cd npm/ruvector
npm publish --access public
cd ../..

echo "All packages published successfully!"

CI/CD Integration (GitHub Actions)

# .github/workflows/publish-npm.yml

name: Publish NPM Packages

on:
  push:
    tags:
      - 'v*'

jobs:
  build-native:
    strategy:
      matrix:
        include:
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
            platform: linux-x64
          - os: ubuntu-latest
            target: aarch64-unknown-linux-gnu
            platform: linux-arm64
          - os: macos-latest
            target: x86_64-apple-darwin
            platform: darwin-x64
          - os: macos-latest
            target: aarch64-apple-darwin
            platform: darwin-arm64
          - os: windows-latest
            target: x86_64-pc-windows-msvc
            platform: win32-x64

    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          target: ${{ matrix.target }}
      - name: Build native
        run: |
          cd crates/ruvector-node
          cargo build --release --target ${{ matrix.target }}
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: native-${{ matrix.platform }}
          path: target/${{ matrix.target }}/release/

  build-wasm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
          target: wasm32-unknown-unknown
      - name: Install wasm-pack
        run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
      - name: Build WASM
        run: ./scripts/build-wasm.sh
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: wasm-build
          path: npm/wasm/

  publish:
    needs: [build-native, build-wasm]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          registry-url: 'https://registry.npmjs.org'
      - name: Download artifacts
        uses: actions/download-artifact@v4
      - name: Setup packages
        run: ./scripts/create-npm-packages.sh
      - name: Publish packages
        run: ./scripts/publish-npm-packages.sh ${{ github.ref_name }}
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

TypeScript Definitions

Shared Interface (index.d.ts)

/**
 * Ruvector - High-performance vector database
 * TypeScript definitions
 */

export interface VectorDatabaseConfig {
  path?: string;
  dimensions: number;
  metric?: 'cosine' | 'euclidean' | 'dotProduct';
  maxElements?: number;
  efConstruction?: number;
  m?: number;
  quantization?: 'none' | 'scalar' | 'product';
  simdEnabled?: boolean;
}

export interface Vector {
  id: string;
  vector: Float32Array | number[];
  metadata?: Record<string, any>;
}

export interface SearchResult {
  id: string;
  score: number;
  metadata?: Record<string, any>;
}

export interface SearchOptions {
  k?: number;
  ef?: number;
  filter?: (metadata: Record<string, any>) => boolean;
}

export interface InsertOptions {
  batch?: boolean;
  skipDuplicates?: boolean;
}

export interface DatabaseStats {
  vectorCount: number;
  dimensions: number;
  metric: string;
  memoryUsage: number;
  indexSize: number;
  quantization: string;
  simdEnabled: boolean;
}

export class VectorDatabase {
  constructor(config: VectorDatabaseConfig);

  /**
   * Create a new collection
   */
  createCollection(name: string, config?: Partial<VectorDatabaseConfig>): Promise<Collection>;

  /**
   * Get existing collection
   */
  getCollection(name: string): Promise<Collection | null>;

  /**
   * List all collections
   */
  listCollections(): Promise<string[]>;

  /**
   * Delete collection
   */
  deleteCollection(name: string): Promise<boolean>;

  /**
   * Get database statistics
   */
  stats(): Promise<DatabaseStats>;

  /**
   * Close database connection
   */
  close(): Promise<void>;
}

export class Collection {
  /**
   * Insert single vector
   */
  insert(id: string, vector: Float32Array | number[], metadata?: Record<string, any>): Promise<void>;

  /**
   * Insert multiple vectors
   */
  insertBatch(vectors: Vector[]): Promise<void>;

  /**
   * Search for similar vectors
   */
  search(query: Float32Array | number[], options?: SearchOptions): Promise<SearchResult[]>;

  /**
   * Get vector by ID
   */
  get(id: string): Promise<Vector | null>;

  /**
   * Update vector
   */
  update(id: string, vector: Float32Array | number[], metadata?: Record<string, any>): Promise<boolean>;

  /**
   * Delete vector
   */
  delete(id: string): Promise<boolean>;

  /**
   * Count vectors
   */
  count(): Promise<number>;

  /**
   * Clear all vectors
   */
  clear(): Promise<void>;
}

/**
 * Get implementation information
 */
export function getImplementation(): {
  type: 'native' | 'wasm';
  platform: string;
  initialized: boolean;
};

/**
 * Initialize ruvector (if not auto-initialized)
 */
export function initialize(): Promise<void>;

/**
 * Create database instance
 */
export function createDatabase(config: VectorDatabaseConfig): Promise<VectorDatabase>;

export default VectorDatabase;

Platform-Specific Augmentations

// @ruvector/core-*/index.d.ts
/// <reference types="./index" />

/**
 * Native implementation specific features
 */
export interface NativeFeatures {
  simdSupport: boolean;
  threadPoolSize: number;
  memoryMapped: boolean;
}

export function getNativeFeatures(): NativeFeatures;
// @ruvector/wasm/index.d.ts
/// <reference types="./index" />

/**
 * WASM implementation specific features
 */
export interface WasmFeatures {
  simdEnabled: boolean;
  simdSupported: boolean;
  workerPoolSize: number;
  indexedDbEnabled: boolean;
}

export function getWasmInfo(): WasmFeatures;

/**
 * Initialize web worker pool
 */
export function initWorkerPool(poolSize?: number): Promise<void>;

/**
 * Enable IndexedDB persistence
 */
export function enableIndexedDb(dbName?: string): Promise<void>;

API Consistency

Consistent Methods Across Implementations

Both NAPI-RS and WASM implementations must provide identical APIs:

// Identical usage for both native and WASM

import { createDatabase } from 'ruvector';

const db = await createDatabase({
  path: './vectors.db',
  dimensions: 384,
  metric: 'cosine'
});

const collection = await db.createCollection('embeddings');

await collection.insert('doc1', new Float32Array([0.1, 0.2, ...]));

const results = await collection.search(queryVector, { k: 10 });

Implementation Differences (Internal Only)

Feature Native (NAPI-RS) WASM
File I/O Direct filesystem IndexedDB
Threading Tokio runtime Web Workers
SIMD Native instructions WASM SIMD
Memory Direct allocation Linear memory

Performance Characteristics

// Performance hints available via API
const impl = getImplementation();

if (impl.type === 'native') {
  // Native: optimized for throughput
  await collection.insertBatch(largeArray);
} else {
  // WASM: optimized for smaller batches
  for (const chunk of chunkArray(largeArray, 100)) {
    await collection.insertBatch(chunk);
  }
}

Implementation Roadmap

Phase 1: Foundation (Week 1-2)

Tasks:

  1. Create /workspaces/ruvector/npm/ directory structure
  2. Set up build scripts for native and WASM
  3. Create platform detection logic
  4. Implement loader mechanism
  5. Write TypeScript definitions

Deliverables:

  • Complete directory structure
  • Working build scripts
  • Platform detection working

Phase 2: Native Packages (Week 2-3)

Tasks:

  1. Set up NAPI-RS builds for all platforms
  2. Create platform-specific package.json files
  3. Test native bindings on each platform
  4. Set up cross-compilation CI/CD

Deliverables:

  • @ruvector/core-* packages for all platforms
  • Working native bindings
  • CI/CD pipeline for builds

Phase 3: WASM Package (Week 3-4)

Tasks:

  1. Build WASM with and without SIMD
  2. Implement worker pool
  3. Add IndexedDB persistence
  4. Optimize WASM size
  5. Test in browsers and Node.js

Deliverables:

  • @ruvector/wasm package
  • Worker support
  • IndexedDB integration
  • Browser compatibility

Phase 4: Main Package (Week 4-5)

Tasks:

  1. Implement platform detection
  2. Create dynamic loader
  3. Add error handling
  4. Write comprehensive tests
  5. Create usage examples

Deliverables:

  • ruvector main package
  • Full test suite
  • Documentation
  • Examples

Phase 5: CLI Package (Week 5-6)

Tasks:

  1. Package CLI binaries
  2. Create npm package structure
  3. Add global installation support
  4. Write CLI documentation

Deliverables:

  • @ruvector/cli package
  • Global CLI commands
  • MCP server binary
  • CLI documentation

Phase 6: Testing & Documentation (Week 6-7)

Tasks:

  1. Integration testing across platforms
  2. Performance benchmarking
  3. Write comprehensive README files
  4. Create migration guide
  5. Prepare release notes

Deliverables:

  • Test reports
  • Benchmark results
  • Complete documentation
  • Release candidate

Phase 7: Publication (Week 7)

Tasks:

  1. Set up npm organization
  2. Publish beta versions
  3. Gather feedback
  4. Fix issues
  5. Publish stable v0.1.1

Deliverables:

  • Published packages on npm
  • Release announcement
  • Migration guide
  • Support channels

ADRs (Architecture Decision Records)

ADR-001: Use Optional Dependencies for Platform-Specific Packages

Status: Accepted

Context: We need to distribute platform-specific native bindings without forcing users to download all platforms.

Decision: Use npm optionalDependencies for platform packages and programmatic fallback to WASM.

Consequences:

  • ✅ Users only download their platform
  • ✅ Graceful degradation to WASM
  • ✅ Smaller install size
  • ⚠️ Requires runtime detection
  • ⚠️ More complex loader logic

Alternatives Considered:

  1. Single package with all binaries (rejected: too large ~40MB)
  2. Manual platform selection (rejected: poor UX)
  3. Postinstall script download (rejected: security concerns)

ADR-002: WASM as Universal Fallback

Status: Accepted

Context: Not all platforms can be supported with native bindings.

Decision: Include @ruvector/wasm as a required dependency with automatic fallback.

Consequences:

  • ✅ Works on any platform
  • ✅ Browser compatibility
  • ✅ No native compilation needed
  • ⚠️ 40-50% slower than native
  • ⚠️ ~1.5MB added to install

Alternatives Considered:

  1. Native-only (rejected: poor platform coverage)
  2. Pure JavaScript (rejected: too slow)
  3. Optional WASM (rejected: fallback must be guaranteed)

ADR-003: Separate CLI Package

Status: Accepted

Context: CLI tools have different distribution requirements than library.

Decision: Create separate @ruvector/cli package with optional dependency on main package.

Consequences:

  • ✅ Users can install library without CLI
  • ✅ CLI can be installed globally
  • ✅ Cleaner separation of concerns
  • ⚠️ Additional package to maintain
  • ⚠️ Version synchronization needed

Alternatives Considered:

  1. Bundle CLI in main package (rejected: bloat)
  2. Separate binary distribution (rejected: npm is standard)

ADR-004: Platform Detection at Runtime

Status: Accepted

Context: Need to determine which native binding to load.

Decision: Implement runtime platform detection with try-catch fallback logic.

Consequences:

  • ✅ Automatic platform selection
  • ✅ Works without configuration
  • ✅ Handles missing binaries gracefully
  • ⚠️ Small runtime overhead on first load
  • ⚠️ Async initialization required

Alternatives Considered:

  1. Build-time detection (rejected: doesn't work for pre-built packages)
  2. User configuration (rejected: poor UX)
  3. Separate imports (rejected: confusing API)

ADR-005: TypeScript Definitions in All Packages

Status: Accepted

Context: TypeScript is widely used and type safety is critical for vector operations.

Decision: Include .d.ts files in all packages with consistent types.

Consequences:

  • ✅ Full TypeScript support
  • ✅ Better developer experience
  • ✅ Type checking for vectors/metadata
  • ⚠️ Must maintain type definitions
  • ⚠️ Sync types across implementations

Alternatives Considered:

  1. Types-only package (rejected: fragmentation)
  2. No types (rejected: poor DX)
  3. Generate from Rust (considered for future)

Appendix

A. Package Size Estimates

Package Uncompressed Gzipped Install Size
ruvector 150 KB 45 KB ~200 KB
@ruvector/core-linux-x64 6.5 MB 2.1 MB ~8 MB
@ruvector/core-linux-arm64 6.2 MB 2.0 MB ~7.5 MB
@ruvector/core-darwin-x64 5.8 MB 1.9 MB ~7 MB
@ruvector/core-darwin-arm64 5.5 MB 1.8 MB ~6.5 MB
@ruvector/core-win32-x64 7.1 MB 2.3 MB ~9 MB
@ruvector/wasm 2.2 MB 1.1 MB ~3 MB
@ruvector/cli 8.5 MB 2.8 MB ~10 MB

B. Compatibility Matrix

Environment Native WASM Notes
Node.js 18+ ✅ ✅ Full support
Node.js 16 ⚠️ ✅ Native may work, WASM guaranteed
Chrome 90+ ❌ ✅ WASM only
Firefox 89+ ❌ ✅ WASM only
Safari 15+ ❌ ✅ WASM only
Edge 90+ ❌ ✅ WASM only
Deno ❌ ✅ WASM only
Bun ✅ ✅ Native preferred

C. Performance Benchmarks

Operation Native WASM (SIMD) WASM (No SIMD)
Insert (single) 1.0x 0.75x 0.45x
Insert (batch 1000) 1.0x 0.80x 0.50x
Search (k=10) 1.0x 0.72x 0.42x
Search (k=100) 1.0x 0.68x 0.38x
Index build 1.0x 0.65x 0.35x

D. Release Checklist

  • All Rust crates build successfully
  • Native bindings built for all platforms
  • WASM packages built (regular + SIMD)
  • TypeScript definitions generated
  • All tests passing
  • Documentation complete
  • Examples tested
  • Benchmarks run
  • CHANGELOG.md updated
  • Version numbers synced
  • npm packages created
  • CI/CD pipeline validated
  • Pre-publish dry run completed
  • Packages published to npm
  • GitHub release created
  • Documentation site updated

Conclusion

This modular npm package architecture for Ruvector provides:

  1. Optimal Performance: Native bindings where available
  2. Universal Compatibility: WASM fallback for all platforms
  3. Developer Experience: Zero-config with TypeScript support
  4. Flexibility: Users can install specific packages if needed
  5. Maintainability: Clear separation of concerns
  6. Scalability: Easy to add new platforms

The architecture balances performance, compatibility, and usability while maintaining a clean and maintainable codebase.

Next Steps: Begin Phase 1 implementation by creating the directory structure and build scripts.