feat(db): static file storage for immutable block/transaction data

[kariy]

Summary

This PR introduces a static file storage layer for immutable, append-only block and transaction data. Heavy values (headers, transactions, receipts, execution traces, state updates) are moved from MDBX B-trees to sequential flat .dat files, while MDBX retains the role of authoritative index — storing offset pointers and all mutable/random-access data.

Motivation

Katana's MDBX database stores 37 tables, but several contain data that is written once and never modified — headers, transactions, receipts, traces, state updates. Storing these large values in MDBX B-trees causes:

• Write amplification: B-tree page splits and copy-on-write overhead for what is fundamentally sequential, append-only data
• MDBX file bloat: Large values (transactions ~200B–10KB, traces ~10–100KB) inflate the MDBX file, reducing OS page cache effectiveness for the hot mutable state tables

Flat file appends eliminate these costs while preserving ACID consistency through MDBX-gated reads.

Architecture

Old design: everything in MDBX

Read:   MDBX get(key) → decompress → value
Write:  MDBX put(key, compress(value)) → B-tree insert → page splits

New design: MDBX pointers + static file data

Read:   MDBX get(key) → StaticFileRef { offset, length }
        → mmap/pread from .dat file → decompress → value

Write:  append(compress(value)) → .dat file → (offset, length)
        → MDBX put(key, StaticFileRef::pointer(offset, length))

MDBX is the single source of truth for what data exists. A reader's MDBX transaction snapshot determines exactly which static file data is visible. Static files have no independent index — they are raw data blobs addressed solely by MDBX pointers.

What moved to static files

Static File	Contents	Size per entry
headers.dat	Compressed block headers	~300–500B
block_state_updates.dat	Compressed state updates	~1–100KB
transactions.dat	Compressed transactions	~200B–10KB
receipts.dat	Compressed receipts	~100–500B
tx_traces.dat	Compressed execution traces	~10–100KB
block_hashes.dat	Block hashes (fixed 32B)	32B
tx_hashes.dat	Tx hashes (fixed 32B)	32B
tx_blocks.dat	Tx-to-block mapping (fixed 8B)	8B

What stays in MDBX

Table	Reason
Headers, BlockStateUpdates, Transactions, Receipts, TxTraces	Now store StaticFileRef pointers (13 bytes) instead of the actual data
BlockBodyIndices	Too small (~10B) — pointer would be larger than the data
BlockNumbers, TxNumbers	Random-key reverse indexes (keyed by hash)
BlockStatusses	Mutable (finality status changes)
BlockHashes, TxHashes, TxBlocks	Also in static files, kept in MDBX as fallback for fork mode
All state/history/trie/class tables	Mutable or random-key access

StaticFileRef — the MDBX-stored pointer

enum StaticFileRef {
    StaticFile { offset: u64, length: u32 },  // 13 bytes — points to .dat file
    Inline(Vec<u8>),                           // compressed data inline in MDBX
}

• Sequential mode (production): data appended to .dat files, MDBX stores StaticFile pointers
• Fork mode (non-sequential block numbers): data stored Inline in MDBX directly, since static file appends require sequential keys

Key Design Decisions

1. MDBX gates all static file reads

Static file reads are not independent — they always go through an MDBX transaction snapshot first. This ensures:

• No stale reads (reader sees a consistent snapshot)
• No phantom reads (if the pointer exists, the data is durable)
• ACID consistency without a separate transaction coordinator

2. Offset pointers in MDBX, not in .idx files

Early iterations stored offsets in separate .idx files alongside .dat files. This was moved to MDBX because .idx files are outside the MDBX transaction boundary — a reader could see an offset written by an uncommitted transaction, breaking snapshot isolation.

3. BlockBodyIndices stays in MDBX

At ~10 bytes per entry, the 13-byte StaticFileRef pointer is larger than the data itself. Benchmarks confirmed +10% read regression with pointer indirection, which disappeared when moved back to direct MDBX storage.

4. No fsync per block

Static files are not fsynced on every write. On crash, MDBX rolls back uncommitted pointers, and orphaned static file data is truncated on next startup. This matches MDBX's own durability model.

5. Fork mode uses inline storage

Fork providers insert blocks at non-sequential numbers. Since static files require sequential appends, fork mode stores compressed data inline in MDBX (StaticFileRef::Inline). The index tables (BlockHashes, TxHashes, TxBlocks) are also written to MDBX in fork mode as fallback.

6. Classes not moved

Contract classes (100KB–2MB JSON) are keyed by ClassHash (random), not sequential numbers. The crash recovery mechanism relies on cursor-last to find the truncation point, which doesn't work for random keys. Classes are also cached by the executor, so the extra I/O hop matters less. Considered for a future iteration.

Crash Recovery

Static file appends happen before the MDBX transaction commits. A crash between these steps leaves orphaned data:

t0: Append to headers.dat        ← on disk
t1: Append to transactions.dat   ← on disk
t2: MDBX put (pointers)          ← uncommitted
t3: --- CRASH ---
t4: MDBX rolls back              ← pointers gone
    Static files have orphaned tail data

On every Db::open(), recover_static_files() reads the last committed MDBX pointer for each table and truncates the corresponding .dat file to offset + length. Fixed-size files are truncated to count * record_size. This is idempotent and runs before any writes.

I/O Optimizations

Optimization	Impact
pread/pwrite	Lock-free reads via pread (no file offset sharing). Writes serialized with pwrite under a mutex.
mmap for reads	Memory-mapped reads from kernel page cache — zero syscall overhead. Falls back to pread for data not yet remapped.
Buffered writes	Small appends accumulate in a 64KB buffer, flushed as a single pwrite at 256KB. Reduces syscalls from ~54/block to a few per batch.
Two-phase batch insert	insert_block_data_batch(): Phase 1 appends all data to static files (sequential I/O), Phase 2 writes all MDBX pointers (grouped B-tree inserts). Pre-sizes buffers based on batch dimensions.
Conditional dual-write removal	BlockHashes, TxHashes, TxBlocks only written to MDBX in fork mode. Sequential mode skips 3 MDBX puts per block.

Builder API

// Production — default settings
let db = DbBuilder::new().write().build("path/to/db")?;

// Custom static file tuning for heavy sync
let sf = StaticFilesBuilder::new()
    .write_buffer_size(1024 * 1024)
    .flush_threshold(4 * 1024 * 1024);
let db = DbBuilder::new().write().static_files(sf).build("path/to/db")?;

// Tests
let db = DbBuilder::new().in_memory().build_ephemeral()?;

Benchmark Results (file-backed)

Writes — per-block vs batch (new)

Workload	per-block	batch	Speedup
100 blocks × 1 tx	12.0 ms	2.9 ms	4.2× faster
100 blocks × 10 txs	30.3 ms	18.7 ms	1.6× faster
500 blocks × 100 txs	1.12 s	907 ms	1.2× faster

Writes — per-block vs MDBX-only baseline

Workload	Baseline (MDBX only)	New (per-block)	Change
100 blocks × 100 txs	201 ms	199 ms	~0%
500 blocks × 100 txs	1.23 s	1.10 s	-11% faster

Reads — vs MDBX-only baseline (500 blocks × 100 txs)

Operation	Baseline	New	Change
latest_number	268 ns	253 ns	-6%
latest_hash	377 ns	307 ns	-19%
header_by_number	489 ns	493 ns	~0%
block_body_indices	287 ns	306 ns	+7%
transaction_by_hash	1.90 µs	1.93 µs	+2%
receipt_by_hash	1.89 µs	1.90 µs	~0%
full_block (100 txs)	164.5 µs	155.0 µs	-6%

Reads are at parity or faster. The write advantage grows with data volume as MDBX B-tree page splits become more expensive while flat file appends stay O(1).

Advantages

• Reduced MDBX write amplification for large immutable values
• Smaller MDBX file → better OS page cache for mutable state tables
• Faster batch writes via two-phase insertion (4.2× for small blocks)
• Sequential disk layout for block/tx data → better read locality
• ACID consistency preserved through MDBX-gated reads
• Configurable via DbBuilder / StaticFilesBuilder

Disadvantages

• Added complexity — two storage backends instead of one
• Per-block writes ~15-25% slower for small blocks due to dual-write overhead (mitigated by batch API)
• BlockBodyIndices reads +7% — pointer indirection overhead for very small values (kept in MDBX to minimize)
• Fork mode stores data inline — no static file benefit for forked providers
• Crash recovery required — orphaned static file data must be truncated on startup
• Not applicable to random-key tables (classes, contract storage) — only sequential append-only data benefits