Getting Started with BitNet Rust

This guide will help you get up and running with BitNet Rust, a pre-alpha implementation of 1-bit neural network inference with real quantized computation. BitNet-rs implements native I2S, TL1, and TL2 quantization kernels for authentic neural network inference.

Installation

Prerequisites

Rust 1.92.0 or later
CUDA 11.8+ (optional, for GPU acceleration)
Python 3.8+ (optional; Python bindings are scaffolded but not yet validated)

Install from crates.io

cargo install bitnet-cli

Build from source

git clone https://github.com/EffortlessMetrics/BitNet-rs.git
cd BitNet-rs
cargo build --release --no-default-features --features cpu

Feature flags

BitNet Rust supports several feature flags for customization:

cpu: Enable CPU inference with SIMD optimizations (includes QK256/GGML I2_S support)
gpu: Enable CUDA GPU acceleration with device-aware quantization
ffi: Enable C++ FFI bridge for cross-validation
crossval: Enable cross-validation against Microsoft BitNet C++

Important: Default features are empty - always specify features explicitly.

Note on Model Support: BitNet-rs now supports GGML I2_S format (QK256) models in pure Rust without requiring FFI or C++ dependencies. The cpu feature includes automatic detection and transparent kernel dispatch for both BitNet native (32-element) and GGML (256-element) I2_S quantization formats.

# Build with CPU support
cargo build --locked --no-default-features --features cpu

# Build with GPU support
cargo build --locked --no-default-features --features gpu

# Build with both CPU and GPU
cargo build --locked --no-default-features --features "cpu,gpu"

Quick Start

Using the CLI with Real GGUF Models

Download a real BitNet model with trained weights:

# Download official Microsoft BitNet model with I2_S quantization
cargo run --no-default-features -p xtask -- download-model \
    --id microsoft/bitnet-b1.58-2B-4T-gguf \
    --file ggml-model-i2_s.gguf

Validate GGUF model and inspect real weights:

# Verify GGUF compatibility and tensor completeness
cargo run -p bitnet-cli -- compat-check \
    models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf

# Inspect real model weights and quantization format
cargo run -p bitnet-cli -- inspect \
    --model models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf

Run inference with real neural network weights:

# Production inference with strict mode (prevents mock fallbacks)
BITNET_STRICT_MODE=1 cargo run --no-default-features -p xtask -- infer \
    --model models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    --tokenizer models/microsoft-bitnet-b1.58-2B-4T-gguf/tokenizer.json \
    --prompt "Explain quantum computing in simple terms" \
    --deterministic

# Stream generation with device-aware quantization
BITNET_STRICT_MODE=1 cargo run --no-default-features -p xtask -- infer \
    --model models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    --tokenizer models/microsoft-bitnet-b1.58-2B-4T-gguf/tokenizer.json \
    --prompt "Write a story about AI and humans working together" \
    --stream

# Performance measurement with realistic expectations
# Performance varies by model and hardware; QK256 uses scalar kernels (~0.1 tok/s for 2B models)
BITNET_DETERMINISTIC=1 BITNET_SEED=42 cargo run --no-default-features -p xtask -- infer \
    --model models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf \
    --tokenizer models/microsoft-bitnet-b1.58-2B-4T-gguf/tokenizer.json \
    --prompt "Benchmark inference performance" \
    --metrics

Using the Rust API

Add BitNet to your Cargo.toml:

[dependencies]
bitnet = "0.2"  # Check crates.io for latest version

Real GGUF model loading with trained weights:

use bitnet::prelude::*;
use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()> {
    // Load real GGUF model with actual trained neural network weights
    let model = BitNetModel::from_file(
        "models/microsoft-bitnet-b1.58-2B-4T-gguf/ggml-model-i2_s.gguf"
    ).await?;

    // Verify real weights were loaded (not mock tensors)
    println!("Loaded {} tensors with {} parameters",
             model.tensor_count(), model.parameter_count());

    // Create inference engine with device-aware backend selection
    // Strict mode prevents mock fallbacks for production use
    std::env::set_var("BITNET_STRICT_MODE", "1");

    let engine = InferenceEngine::builder()
        .model(model)
        .backend(Backend::Auto)  // Automatically selects GPU (runtime detection)
        .quantization(QuantizationType::I2S)  // Use I2_S quantization
        .strict_mode(true)  // Prevent mock inference fallbacks
        .build()?;

    // Configure generation with performance metrics
    let config = GenerationConfig {
        max_new_tokens: 100,
        temperature: 0.7,
        enable_metrics: true,  // Track performance
        ..Default::default()
    };

    // Generate text with real neural network inference
    let response = engine.generate_with_config(
        "Explain the benefits of 1-bit neural networks",
        &config
    ).await?;

    println!("Generated: {}", response.text);

    // Access performance metrics from real quantized inference
    // Performance varies by model, quantization format, and hardware
    if let Some(metrics) = response.metrics {
        println!("Inference time: {:.2}ms", metrics.timing.total);
        println!("Throughput: {:.1} tokens/sec", metrics.throughput.e2e);
        println!("Quantization: {}", metrics.quantization_type);
        println!("Device: {}", metrics.device_info);
    }

    Ok(())
}

Streaming Generation

use bitnet::prelude::*;
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<()> {
    // Load GGUF model
    let model = BitNetModel::from_file("model.gguf").await?;

    // Create inference engine
    let engine = InferenceEngine::builder()
        .model(model)
        .backend(Backend::Auto)
        .build()?;

    // Configure streaming generation
    let config = GenerationConfig {
        max_new_tokens: 100,
        temperature: 0.7,
        ..Default::default()
    };

    let mut stream = engine.generate_stream_with_config("Tell me a story", &config);

    while let Some(result) = stream.next().await {
        match result {
            Ok(response) => print!("{}", response.text),
            Err(e) => eprintln!("Error: {}", e),
        }
    }

    Ok(())
}

Model Formats

BitNet Rust supports multiple model formats:

GGUF Format

# Load GGUF model
bitnet-cli inference --model path/to/model.gguf --prompt "Hello"

SafeTensors Format

# Load SafeTensors model
bitnet-cli inference --model path/to/model.safetensors --prompt "Hello"

HuggingFace Directory

# Load a local HuggingFace model directory
bitnet-cli inference --model path/to/hf-model --prompt "Hello"

HuggingFace Hub

# Load from HuggingFace Hub
bitnet-cli inference --model microsoft/bitnet-b1_58-large --prompt "Hello"

Configuration

Configuration File

Create a bitnet.toml configuration file:

[model]
default_model = "microsoft/bitnet-b1_58-large"
cache_dir = "~/.cache/bitnet"

[inference]
device = "auto"  # "cpu", "cuda", or "auto"
max_batch_size = 8
kv_cache_size = 2048

[generation]
max_new_tokens = 512
temperature = 0.7
top_p = 0.9
top_k = 50
repetition_penalty = 1.0

Environment Variables

BitNet Rust respects these environment variables:

BITNET_MODEL_CACHE: Model cache directory
BITNET_DEVICE: Default device ("cpu", "cuda", "auto")
BITNET_LOG_LEVEL: Log level ("trace", "debug", "info", "warn", "error")
BITNET_STRICT_MODE: Prevent mock inference fallbacks ("1" enables strict mode)
BITNET_DETERMINISTIC: Enable deterministic inference for reproducible results
BITNET_SEED: Set seed for reproducible inference (works with BITNET_DETERMINISTIC=1)
CUDA_VISIBLE_DEVICES: GPU device selection

Performance Optimization

Performance Expectations (Issue #260 Completed)

BitNet-rs provides realistic performance baselines based on real quantized computation without mock fallbacks:

CPU Performance: Varies by model and hardware. QK256 models use scalar kernels (~0.1 tok/s for 2B). I2_S BitNet32-F16 is significantly faster with SIMD optimization.
GPU Performance: GPU backends are alpha/scaffolded. Performance not yet measured.
Quantization Accuracy: Accuracy targets defined in test fixtures. Formal measurement infrastructure is pending.
Strict Mode: Use BITNET_STRICT_MODE=1 to prevent any mock fallbacks

CPU Optimization

Enable CPU features with strict mode:

RUSTFLAGS="-C target-cpu=native" cargo build --release --no-default-features --features cpu
BITNET_STRICT_MODE=1 bitnet-cli inference --model model.gguf --prompt "Hello"

Tune thread count for optimal performance:

export RAYON_NUM_THREADS=8
BITNET_STRICT_MODE=1 bitnet-cli inference --model model.gguf --prompt "Hello"

GPU Optimization

Note: GPU backends are scaffolded (alpha). The examples below show the intended API.

Enable mixed precision with strict mode:

// Enable strict mode to prevent mock GPU fallbacks
std::env::set_var("BITNET_STRICT_MODE", "1");

let config = InferenceConfig {
    use_mixed_precision: true,
    precision_mode: PrecisionMode::Auto,  // FP16/BF16 based on device capability
    device_aware: true,  // Enable device-aware quantization selection
    ..Default::default()
};

Optimize batch size and validate GPU usage:

let config = InferenceConfig {
    max_batch_size: 16,  // Adjust based on GPU memory
    enable_gpu_validation: true,  // Validate GPU kernel execution
    ..Default::default()
};

Troubleshooting

Common Issues

CUDA not found:
- Install CUDA 11.8 or later
- Set CUDA_HOME environment variable
- Build with --no-default-features --features cli for CPU-only
Model loading fails:
- Check model format compatibility
- Verify model file integrity
- Ensure sufficient disk space and memory
Poor performance:
- CRITICAL: Verify strict mode is enabled: BITNET_STRICT_MODE=1 to prevent mock fallbacks
- Check for mock inference warnings in logs (should be eliminated in Issue #260)
- Enable native CPU features with RUSTFLAGS="-C target-cpu=native"
- Use GPU acceleration (compiled with gpu feature)
- Adjust batch size and thread count
- Performance depends on model, quantization format, and hardware

Debug Mode

Enable debug logging:

RUST_LOG=debug bitnet-cli inference --model model.gguf --prompt "Hello"

Memory Issues

Monitor memory usage:

bitnet-cli benchmark --model model.gguf --monitor-memory

Next Steps

Read the API Reference for detailed API documentation
Check out Examples for more usage patterns
See Migration Guide for migrating from Python/C++
Review Performance Tuning for optimization tips

Navigation Menu

Uh oh!

Getting Started with BitNet Rust

Installation

Prerequisites

Install from crates.io

Build from source

Feature flags

Quick Start

Using the CLI with Real GGUF Models

Using the Rust API

Streaming Generation

Model Formats

GGUF Format

SafeTensors Format

HuggingFace Directory

HuggingFace Hub

Configuration

Configuration File

Environment Variables

Performance Optimization

Performance Expectations (Issue #260 Completed)

CPU Optimization

GPU Optimization

Troubleshooting

Common Issues

Debug Mode

Memory Issues

Next Steps

Getting Help

Uh oh!

FilesExpand file tree

getting-started.md

Latest commit

History

getting-started.md

File metadata and controls

Getting Started with BitNet Rust

Installation

Prerequisites

Install from crates.io

Build from source

Feature flags

Quick Start

Using the CLI with Real GGUF Models

Using the Rust API

Streaming Generation

Model Formats

GGUF Format

SafeTensors Format

HuggingFace Directory

HuggingFace Hub

Configuration

Configuration File

Environment Variables

Performance Optimization

Performance Expectations (Issue #260 Completed)

CPU Optimization

GPU Optimization

Troubleshooting

Common Issues

Debug Mode

Memory Issues

Next Steps

Getting Help