Implement DB benchmark #18117
Implement DB benchmark #18117
Conversation
Result of the run with default parameters
📊 Benchmark Analysis: Write vs Read PerformanceTest Configuration:
✍️ Write Performance ComparisonSpeed (Time/Run - lower is better):
Memory Allocation:
📖 Read Performance ComparisonSpeed (Time/Run - all very fast):
All read operations are extremely fast (microsecond range vs millisecond writes). 🎯 Key Takeaways✅ LMDB: Terrible write performance (~2.7s per operation) but excellent read speed 💡 Recommendation: For large-value workloads like this (128 KB per value):
Update after optimization of multi-file writing📊 multi_file_write Benchmark Results
📈 Before vs After ComparisonPerformance Gains:
🏆 Updated Write Performance Rankings
💡 What Changed?The massive mjWd reduction (from 2M+ to 24w) suggests you eliminated file system churn or excessive allocations. This is a textbook example of optimization - you kept the speed advantage while making it vastly more GC-friendly. New recommendation: For large-value (128 KB) write workloads, |
🚀 Smaller Values, Different Story📊 Full Benchmark Results (New Parameters)Test Configuration:
Before vs After:
🏆 Write Performance Rankings (8.8 KB values)
📖 Read Performance Rankings
💡 Key ObservationsCompared to 128 KB value test:
Optimization trade-off: The optimized version trades some speed for much better GC behavior. Depending on workload (GC pressure vs raw throughput), either version could be preferable. |
| Sys.getenv "WARMUP_BLOCKS" |> Option.value_map ~default:800 ~f:int_of_string | ||
|
|
||
| (* Fixed seed for reproducibility *) | ||
| let random_seed = 42 |
There was a problem hiding this comment.
Would be nice to make this random if not user provided. And the test prints out the seed it's using every time.
| let cached_value = lazy (generate_value ()) | ||
|
|
||
| (* Get the cached value *) | ||
| let get_value () = Lazy.force cached_value |
There was a problem hiding this comment.
We're using a same value for all blocks. I wonder if these backend would have some optimizations that make the performance better, it's better to use distinct values for distinct keys.
| min_key + Random.State.int random_state (max_key - min_key + 1) | ||
|
|
||
| (* Database interface that all implementations must satisfy *) | ||
| module type Database = sig |
There was a problem hiding this comment.
It might be worth replace this implementation that stores string with implementation storing bytes to avoid any kind of wrappings that'll be done at bindings, end, so that we're not wasting time to do serialization/deserialization.
| let start_key = block_num * Common.keys_per_block in | ||
| List.iteri values ~f:(fun i value -> | ||
| let key = start_key + i in | ||
| Rw.set ~env:t.env t.db key value ) |
There was a problem hiding this comment.
We're wasting time to commit on each key in the block. Is this intended?
| let path = block_path t block_num in | ||
| (* Write all values directly to file without in-memory concatenation *) | ||
| Out_channel.with_file path ~binary:true ~f:(fun oc -> | ||
| List.iter values ~f:(Out_channel.output_string oc) ) |
There was a problem hiding this comment.
This is not fair competition. Files are written to a same handler in a same channel without close/open.
But say with LMDB, we're creating a txn per write.
| let set_block db ~block_num values = | ||
| let start_key = block_num * Common.keys_per_block in | ||
| List.iteri values ~f:(fun i value -> | ||
| let key = start_key + i in |
glyh
left a comment
glyh
left a comment
There was a problem hiding this comment.
Please consider use batch operations on DB to the very least
This PR implements a benchmark for DB usage.
It independently measures read and write performance of every DB imnplementation.
Allows to make informed decisions for various flows of working with data, keeping measurement of pure DB performance separate from performance of other subsystems (including serialization).
See db_benchmark/README.md for more details about the benchmark.
Explain how you tested your changes:
Checklist: