Skip to content

Roll CoinStore, BlockStore and BlockHeightMap into ConsensusStore #19949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Roll CoinStore, BlockStore and BlockHeightMap into ConsensusStore #19949

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0da0073
`ConsensusStore`
richardkiss Aug 12, 2025
e853f48
ConsensusStoreSQLite3
richardkiss Aug 13, 2025
5cd7fb9
fix tests
richardkiss Aug 13, 2025
30a2e63
more tests
richardkiss Aug 13, 2025
b91960c
Move `coin_store_protocol.py` to `chia/full_node`
richardkiss Aug 13, 2025
b238338
tach
richardkiss Aug 14, 2025
4abbe5c
first crack at async context manager for ConsensusStore
richardkiss Aug 18, 2025
6706d6a
Handle nested context manager
richardkiss Aug 18, 2025
4f19c4c
More protocol
richardkiss Aug 18, 2025
9f1447b
more
richardkiss Aug 18, 2025
fda5a08
mypy
richardkiss Aug 18, 2025
081ba0e
mypy
richardkiss Aug 18, 2025
f742272
ruff
richardkiss Aug 18, 2025
00f76ce
simplify
richardkiss Aug 25, 2025
15f0839
simplify
richardkiss Aug 25, 2025
65642be
dataclass
richardkiss Aug 25, 2025
e1c728e
CoinStoreProtocol
richardkiss Aug 26, 2025
034018b
shrink diff
richardkiss Aug 26, 2025
b7499a8
Undo test changes and silly api
richardkiss Sep 5, 2025
246f3c6
Defer to `DBWrapper2`
richardkiss Sep 5, 2025
24ad6d4
coverage
richardkiss Sep 5, 2025
9f4f99b
Factor `_load_chain_from_store` into pieces
richardkiss Sep 6, 2025
eafd0c4
Don't use double underscore.
richardkiss Sep 6, 2025
c285715
simplify diffs
richardkiss Sep 8, 2025
f617564
minimize diffs
richardkiss Sep 8, 2025
a6c9748
more minimize diffs
richardkiss Sep 8, 2025
5e394bb
Remove comments
richardkiss Sep 11, 2025
fd1edd5
Self
richardkiss Sep 11, 2025
4f7e799
final
richardkiss Sep 17, 2025
acfd7d5
Remove `ConsensusStoreWriterSQLite3.writer` method
richardkiss Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion benchmarks/block_ref.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from chia.consensus.get_block_generator import get_block_generator
from chia.full_node.block_store import BlockStore
from chia.full_node.coin_store import CoinStore
from chia.full_node.consensus_store_sqlite3 import ConsensusStoreSQLite3
from chia.types.blockchain_format.serialized_program import SerializedProgram
from chia.util.db_version import lookup_db_version
from chia.util.db_wrapper import DBWrapper2
Expand Down Expand Up @@ -70,7 +71,8 @@ async def main(db_path: Path) -> None:
# make configurable
reserved_cores = 4
height_map = await BlockHeightMap.create(db_path.parent, db_wrapper)
blockchain = await Blockchain.create(coin_store, block_store, height_map, DEFAULT_CONSTANTS, reserved_cores)
consensus_store = ConsensusStoreSQLite3(block_store, coin_store, height_map)
blockchain = await Blockchain.create(consensus_store, DEFAULT_CONSTANTS, reserved_cores)

peak = blockchain.get_peak()
assert peak is not None
Expand Down
7 changes: 4 additions & 3 deletions chia/_tests/blockchain/blockchain_test_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,20 @@
from chia.consensus.blockchain import AddBlockResult, Blockchain
from chia.consensus.difficulty_adjustment import get_next_sub_slot_iters_and_difficulty
from chia.consensus.multiprocess_validation import PreValidationResult, pre_validate_block
from chia.full_node.consensus_store_sqlite3 import ConsensusStoreSQLite3
from chia.types.validation_state import ValidationState
from chia.util.errors import Err


async def check_block_store_invariant(bc: Blockchain):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function checks the invariant of the sqlite database. I don't think it makes sense to make it generic to check the invariant of any block store. The right hand, generic, version looks a bit odd. get_block_heights_in_main_chain() sounds like it would return range(0, peak.height).

That's not really what this invariant check is ensuring. I think you should leave this function as-is, but only operate on the sqlite block store. The future RocksDB block store will most likely have different invariants to check

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like a very specific invariant, one that should probably be true for any block store that supports in_main_chain (and I'm actually quite skeptical about the usefulness of keeping and constantly updating this flag, since it's a bit redundant with the in-RAM BlockHeightMap). I'm fine returning this test to its previous state and removing the rather silly API I created.

The test will have to check that bc.block_store actually exists (and when it's eventually replaced with BlockArchiveStore, have dig down to get the implementation details).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is now much closer to its original form.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and I'm actually quite skeptical about the usefulness of keeping and constantly updating this flag, since it's a bit redundant with the in-RAM BlockHeightMap)

The in RAM map is not sustainable and will need to be removed eventually

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the api to ConsensusStore seems unreasonably large, with a lot of redundancy. It could be an interesting future project to simplify the api; we could, at the same time, remove the annoying non-async methods that BlockHeightMap currently provides, and that caching moved to either FullNode or ConsensusStore.

db_wrapper = bc.block_store.db_wrapper
assert isinstance(bc.consensus_store, ConsensusStoreSQLite3)

if db_wrapper.db_version == 1:
if bc.consensus_store.block_store.db_wrapper == 1:
return

in_chain = set()
max_height = -1
async with bc.block_store.transaction() as conn:
async with bc.consensus_store.block_store.transaction() as conn:
async with conn.execute("SELECT height, in_main_chain FROM full_blocks") as cursor:
rows = await cursor.fetchall()
for row in rows:
Expand Down
23 changes: 12 additions & 11 deletions chia/_tests/blockchain/test_blockchain.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2076,7 +2076,7 @@ async def test_timelock_conditions(

if expected == AddBlockResult.NEW_PEAK:
# ensure coin was in fact spent
c = await b.coin_store.get_coin_record(coin.name())
c = await b.consensus_store.get_coin_record(coin.name())
assert c is not None and c.spent

@pytest.mark.anyio
Expand Down Expand Up @@ -2286,10 +2286,10 @@ async def test_ephemeral_timelock(

if expected == AddBlockResult.NEW_PEAK:
# ensure coin1 was in fact spent
c = await b.coin_store.get_coin_record(coin1.name())
c = await b.consensus_store.get_coin_record(coin1.name())
assert c is not None and c.spent
# ensure coin2 was NOT spent
c = await b.coin_store.get_coin_record(coin2.name())
c = await b.consensus_store.get_coin_record(coin2.name())
assert c is not None and not c.spent

@pytest.mark.anyio
Expand Down Expand Up @@ -3103,9 +3103,9 @@ async def test_double_spent_in_reorg(self, empty_blockchain: Blockchain, bt: Blo
)

# ephemeral coin is spent
first_coin = await b.coin_store.get_coin_record(new_coin.name())
first_coin = await b.consensus_store.get_coin_record(new_coin.name())
assert first_coin is not None and first_coin.spent
second_coin = await b.coin_store.get_coin_record(tx_2.additions()[0].name())
second_coin = await b.consensus_store.get_coin_record(tx_2.additions()[0].name())
assert second_coin is not None and not second_coin.spent

farmer_coin = create_farmer_coin(
Expand All @@ -3121,7 +3121,7 @@ async def test_double_spent_in_reorg(self, empty_blockchain: Blockchain, bt: Blo
)
await _validate_and_add_block(b, blocks_reorg[-1])

farmer_coin_record = await b.coin_store.get_coin_record(farmer_coin.name())
farmer_coin_record = await b.consensus_store.get_coin_record(farmer_coin.name())
assert farmer_coin_record is not None and farmer_coin_record.spent

@pytest.mark.anyio
Expand Down Expand Up @@ -3876,11 +3876,12 @@ async def test_chain_failed_rollback(empty_blockchain: Blockchain, bt: BlockTool
await _validate_and_add_block(b, block, expected_result=AddBlockResult.ADDED_AS_ORPHAN, fork_info=fork_info)

# Incorrectly set the height as spent in DB to trigger an error
print(f"{await b.coin_store.get_coin_record(spend_bundle.coin_spends[0].coin.name())}")
print(f"{await b.consensus_store.get_coin_record(spend_bundle.coin_spends[0].coin.name())}")
print(spend_bundle.coin_spends[0].coin.name())
# await b.coin_store._set_spent([spend_bundle.coin_spends[0].coin.name()], 8)
await b.coin_store.rollback_to_block(2)
print(f"{await b.coin_store.get_coin_record(spend_bundle.coin_spends[0].coin.name())}")
# await b.consensus_store._set_spent([spend_bundle.coin_spends[0].coin.name()], 8)
async with b.consensus_store.writer() as writer:
await writer.rollback_to_block(2)
print(f"{await b.consensus_store.get_coin_record(spend_bundle.coin_spends[0].coin.name())}")

fork_block = blocks_reorg_chain[10 - 1]
# fork_info = ForkInfo(fork_block.height, fork_block.height, fork_block.header_hash)
Expand Down Expand Up @@ -4209,7 +4210,7 @@ async def get_fork_info(blockchain: Blockchain, block: FullBlock, peak: BlockRec
counter = 0
start = time.monotonic()
for height in range(fork_info.fork_height + 1, block.height):
fork_block: Optional[FullBlock] = await blockchain.block_store.get_full_block(fork_chain[uint32(height)])
fork_block: Optional[FullBlock] = await blockchain.consensus_store.get_full_block(fork_chain[uint32(height)])
assert fork_block is not None
assert fork_block.height - 1 == fork_info.peak_height
assert fork_block.height == 0 or fork_block.prev_header_hash == fork_info.peak_hash
Expand Down
10 changes: 3 additions & 7 deletions chia/_tests/core/full_node/ram_db.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,8 @@

from chia_rs import ConsensusConstants

from chia.consensus.block_height_map import BlockHeightMap
from chia.consensus.blockchain import Blockchain
from chia.full_node.block_store import BlockStore
from chia.full_node.coin_store import CoinStore
from chia.full_node.consensus_store_sqlite3 import ConsensusStoreSQLite3
from chia.util.db_wrapper import DBWrapper2


Expand All @@ -20,10 +18,8 @@ async def create_ram_blockchain(
) -> AsyncIterator[tuple[DBWrapper2, Blockchain]]:
uri = f"file:db_{random.randint(0, 99999999)}?mode=memory&cache=shared"
async with DBWrapper2.managed(database=uri, uri=True, reader_count=1, db_version=2) as db_wrapper:
block_store = await BlockStore.create(db_wrapper)
coin_store = await CoinStore.create(db_wrapper)
height_map = await BlockHeightMap.create(Path("."), db_wrapper)
blockchain = await Blockchain.create(coin_store, block_store, height_map, consensus_constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, Path("."))
blockchain = await Blockchain.create(consensus_store, consensus_constants, 2)
try:
yield db_wrapper, blockchain
finally:
Expand Down
85 changes: 34 additions & 51 deletions chia/_tests/core/full_node/stores/test_block_store.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
from chia._tests.blockchain.blockchain_test_utils import _validate_and_add_block
from chia._tests.util.db_connection import DBConnection, PathDBConnection
from chia.consensus.block_body_validation import ForkInfo
from chia.consensus.block_height_map import BlockHeightMap
from chia.consensus.blockchain import AddBlockResult, Blockchain
from chia.consensus.default_constants import DEFAULT_CONSTANTS
from chia.consensus.full_block_to_block_record import header_block_to_sub_block_record
from chia.full_node.block_store import BlockStore
from chia.full_node.coin_store import CoinStore
from chia.full_node.consensus_store_sqlite3 import ConsensusStoreSQLite3
from chia.full_node.full_block_utils import GeneratorBlockInfo
from chia.simulator.block_tools import BlockTools
from chia.simulator.wallet_tools import WalletTool
Expand Down Expand Up @@ -71,10 +70,8 @@ async def test_block_store(tmp_dir: Path, db_version: int, bt: BlockTools, use_c

async with DBConnection(db_version) as db_wrapper, DBConnection(db_version) as db_wrapper_2:
# Use a different file for the blockchain
coin_store_2 = await CoinStore.create(db_wrapper_2)
store_2 = await BlockStore.create(db_wrapper_2, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper_2)
bc = await Blockchain.create(coin_store_2, store_2, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper_2, tmp_dir, use_cache=use_cache)
bc = await Blockchain.create(consensus_store, bt.constants, 2)

store = await BlockStore.create(db_wrapper, use_cache=use_cache)
await BlockStore.create(db_wrapper_2)
Expand Down Expand Up @@ -147,10 +144,9 @@ async def test_get_full_blocks_at(

async with DBConnection(2) as db_wrapper:
# Use a different file for the blockchain
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

count = 0
fork_info = ForkInfo(-1, -1, bt.constants.GENESIS_CHALLENGE)
Expand All @@ -175,10 +171,9 @@ async def test_get_block_records_in_range(

async with DBConnection(2) as db_wrapper:
# Use a different file for the blockchain
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

count = 0
fork_info = ForkInfo(-1, -1, bt.constants.GENESIS_CHALLENGE)
Expand All @@ -205,10 +200,9 @@ async def test_get_block_bytes_in_range_in_main_chain(

async with DBConnection(2) as db_wrapper:
# Use a different file for the blockchain
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)
count = 0
fork_info = ForkInfo(-1, -1, bt.constants.GENESIS_CHALLENGE)
for b1, b2 in zip(blocks, alt_blocks):
Expand All @@ -234,10 +228,8 @@ async def test_deadlock(tmp_dir: Path, db_version: int, bt: BlockTools, use_cach

async with PathDBConnection(db_version) as wrapper, PathDBConnection(db_version) as wrapper_2:
store = await BlockStore.create(wrapper, use_cache=use_cache)
coin_store_2 = await CoinStore.create(wrapper_2)
store_2 = await BlockStore.create(wrapper_2)
height_map = await BlockHeightMap.create(tmp_dir, wrapper_2)
bc = await Blockchain.create(coin_store_2, store_2, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(wrapper_2, tmp_dir, use_cache=use_cache)
bc = await Blockchain.create(consensus_store, bt.constants, 2)
block_records = []
for block in blocks:
await _validate_and_add_block(bc, block)
Expand Down Expand Up @@ -265,10 +257,9 @@ async def test_rollback(bt: BlockTools, tmp_dir: Path, use_cache: bool, default_

async with DBConnection(2) as db_wrapper:
# Use a different file for the blockchain
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

# insert all blocks
count = 0
Expand Down Expand Up @@ -328,10 +319,9 @@ async def test_count_compactified_blocks(bt: BlockTools, tmp_dir: Path, db_versi
blocks = bt.get_consecutive_blocks(10)

async with DBConnection(db_version) as db_wrapper:
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

count = await block_store.count_compactified_blocks()
assert count == 0
Expand All @@ -349,10 +339,9 @@ async def test_count_uncompactified_blocks(bt: BlockTools, tmp_dir: Path, db_ver
blocks = bt.get_consecutive_blocks(10)

async with DBConnection(db_version) as db_wrapper:
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

count = await block_store.count_uncompactified_blocks()
assert count == 0
Expand All @@ -377,10 +366,9 @@ def rand_vdf_proof() -> VDFProof:
)

async with DBConnection(db_version) as db_wrapper:
coin_store = await CoinStore.create(db_wrapper)
block_store = await BlockStore.create(db_wrapper, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper)
bc = await Blockchain.create(coin_store, block_store, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper, tmp_dir, use_cache=use_cache)
block_store = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)
for block in blocks:
await _validate_and_add_block(bc, block)

Expand All @@ -400,7 +388,7 @@ def rand_vdf_proof() -> VDFProof:

# make sure we get the same result when we hit the database
# itself (and not just the block cache)
block_store.rollback_cache_block(block.header_hash)
consensus_store.rollback_cache_block(block.header_hash)
b = await block_store.get_full_block(block.header_hash)
assert b is not None
assert b.challenge_chain_ip_proof == proof
Expand Down Expand Up @@ -458,10 +446,8 @@ async def test_get_blocks_by_hash(tmp_dir: Path, bt: BlockTools, db_version: int

async with DBConnection(db_version) as db_wrapper, DBConnection(db_version) as db_wrapper_2:
# Use a different file for the blockchain
coin_store_2 = await CoinStore.create(db_wrapper_2)
store_2 = await BlockStore.create(db_wrapper_2, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper_2)
bc = await Blockchain.create(coin_store_2, store_2, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper_2, tmp_dir, use_cache=use_cache)
bc = await Blockchain.create(consensus_store, bt.constants, 2)

store = await BlockStore.create(db_wrapper, use_cache=use_cache)
await BlockStore.create(db_wrapper_2)
Expand Down Expand Up @@ -498,10 +484,9 @@ async def test_get_block_bytes_in_range(tmp_dir: Path, bt: BlockTools, db_versio

async with DBConnection(db_version) as db_wrapper_2:
# Use a different file for the blockchain
coin_store_2 = await CoinStore.create(db_wrapper_2)
store_2 = await BlockStore.create(db_wrapper_2, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper_2)
bc = await Blockchain.create(coin_store_2, store_2, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper_2, tmp_dir, use_cache=use_cache)
store_2 = consensus_store.block_store
bc = await Blockchain.create(consensus_store, bt.constants, 2)

await BlockStore.create(db_wrapper_2)

Expand Down Expand Up @@ -571,10 +556,8 @@ async def test_get_prev_hash(tmp_dir: Path, bt: BlockTools, db_version: int, use

async with DBConnection(db_version) as db_wrapper, DBConnection(db_version) as db_wrapper_2:
# Use a different file for the blockchain
coin_store_2 = await CoinStore.create(db_wrapper_2)
store_2 = await BlockStore.create(db_wrapper_2, use_cache=use_cache)
height_map = await BlockHeightMap.create(tmp_dir, db_wrapper_2)
bc = await Blockchain.create(coin_store_2, store_2, height_map, bt.constants, 2)
consensus_store = await ConsensusStoreSQLite3.create(db_wrapper_2, tmp_dir, use_cache=use_cache)
bc = await Blockchain.create(consensus_store, bt.constants, 2)

store = await BlockStore.create(db_wrapper, use_cache=use_cache)
await BlockStore.create(db_wrapper_2)
Expand Down
Loading
Loading