Why is RocksDB slower when using app_data?

[laleksiunas]

I have a RocksDB setup like this:

use actix_web::{get, web, App, HttpResponse, HttpServer, Responder, error};
use rocksdb::DB;
use std::sync::Arc;
use env_logger;

#[derive(Clone)]
struct TestDb {
    db: Arc<DB>,
}

#[get("/from-data")]
async fn test_from_data(db: web::Data<TestDb>) -> actix_web::Result<impl Responder> {
    let now = std::time::Instant::now();

    db.db.put(b"test-key", b"test-value").unwrap();

    println!("From data elapsed: {:.2?}", now.elapsed());

    Ok(HttpResponse::Ok())
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    std::env::set_var("RUST_LOG", "debug");

    env_logger::init();

    let db = TestDb { db: Arc::new(DB::open_default("./rocks.db").unwrap()) };

    HttpServer::new(move || {
        App::new()
            .app_data(web::Data::new(db.clone()))
            .service(test_from_data)
    })
        .bind(("127.0.0.1", 8080))?
        .run()
        .await
}

App works fine, but it yields unexpected results:
From data elapsed: 12.12ms
From data elapsed: 17.96ms
From data elapsed: 18.08ms
From data elapsed: 15.66ms
From data elapsed: 54.60µs
From data elapsed: 17.40ms
From data elapsed: 2.33ms
From data elapsed: 16.71ms
From data elapsed: 16.23ms
From data elapsed: 18.42ms
From data elapsed: 16.76ms
From data elapsed: 53.70µs
From data elapsed: 8.78ms
From data elapsed: 15.86ms
From data elapsed: 18.35ms
From data elapsed: 44.64µs
Most of the time database read takes >10ms, however, there are times like 54.60µs, 44.64µs as well.

Then, if I open the database in route handler directly, I get really different results.

#[get("/from-opened")]
async fn test_from_opened() -> actix_web::Result<impl Responder> {
    let db = TestDb { db: Arc::new(DB::open_default("./rocks.db").unwrap()) };

    let now = std::time::Instant::now();

    db.db.put(b"test-key", b"test-value").unwrap();

    println!("From opened elapsed: {:.2?}", now.elapsed());

    Ok(HttpResponse::Ok())
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    std::env::set_var("RUST_LOG", "debug");

    env_logger::init();

    HttpServer::new(move || {
        App::new()
            .service(test_from_opened)
    })
        .bind(("127.0.0.1", 8080))?
        .run()
        .await
}

From opened elapsed: 78.07µs
From opened elapsed: 52.26µs
From opened elapsed: 52.99µs
From opened elapsed: 37.00µs
From opened elapsed: 46.59µs
From opened elapsed: 39.55µs
From opened elapsed: 55.33µs
From opened elapsed: 73.69µs
From opened elapsed: 45.08µs
From opened elapsed: 59.33µs
From opened elapsed: 38.24µs
From opened elapsed: 46.16µs
From opened elapsed: 95.77µs
From opened elapsed: 38.39µs
From opened elapsed: 68.75µs
From opened elapsed: 42.86µs
From opened elapsed: 50.75µs
From opened elapsed: 51.62µs
From opened elapsed: 53.53µs
From opened elapsed: 50.10µs

Why is that? I tried putting DB instance as a static variable using lazy_static but it still did not solve performance issue. I cannot open the DB every time, because open takes 60ms on its own.

Thank you.

Running the app using:
cargo run --release

Environment:

actix-web = "4.3.1"
rocksdb = "0.20.1"
env_logger = "0.10.0"
serde = { version = "1.0.160", features = ["derive"] }

[enzious]

I think the issue is you're sharing the first db across all threads, which might be blocking internally on put? But also blocking in your actix-web handler.

Try a main something like this (this still blocks, but not all workers):

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    std::env::set_var("RUST_LOG", "debug");

    env_logger::init();

    HttpServer::new(move || {
        // one db per worker
        let db = TestDb { db: Arc::new(DB::open_default("./rocks.db").unwrap()) };

        App::new()
            .app_data(web::Data::new(db))
            .service(test_from_data)
    })
        .bind(("127.0.0.1", 8080))?
        .run()
        .await
}

[RoDmitry]

Maybe try using OnceLock in static, instead of lazy_static.
I think it might be an optimization to take place here. Compiler optimized Arc out.