feat: Add block database #1340
Draft
feat: Add block database #1340
+2,098
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DracoLi
force-pushed
the
dl/block-database
branch
from
1361e38 to
989a5fd
Compare
DracoLi
force-pushed
the
dl/block-database
branch
from
989a5fd to
b4fa362
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why this should be merged
Introduces a new
ethdb.Databasewrapper that stores block headers, bodies, and receipts in separate height-indexed databases, moving away from storing all block data in the main key-value database.This should reduce database compactions in the key-value store, leading to improved performance for all existing read/write operations.
There's a migrator that will migrate block data from the key-value database to the block database in the background.
See https://github.com/ava-labs/avalanchego/blob/master/x/blockdb/README.md on how the height-indexed database works.
How this works
The
BlockDatabasewrapper intercepts database operations and routes block-related keys (headers, bodies, receipts) to dedicated height-indexed databases while passing all other operations through to the underlying key-value database.Block headers and bodies are stored together in a single entry, requiring the system to wait for both components before writing to the block database.
Genesis block (block 0) is not stored in the block database to simplify implementation.
During the migration period, the wrapper falls back to the key-value database for any data not yet migrated, ensuring zero downtime and data availability throughout the transition.
How this was tested
Unit tests
Benchmark
Benchmarked using the Reexecute Benchmark across multiple block ranges:
Mainnet Nodes
Default Mainnet node running blockdb for +4 weeks without issues
Migration Testing
The following scenarios are tested on Mainnet nodes:
Block Data Verification
Created a script that took two evm rpc endpoints and periodically check the block and receipts data return by those two endpoints for blocks from the last checked height to the latest block height to ensure they are the same.
This script is running 24/7 and is monitoring a blockdb node and a node running v1.13.5.