Differentiate between Consensus and Cluster Headers storage #8222

tim-barry · 2025-12-04T00:08:37Z

A ChainID must now be provided to Headers storage instance on creation.
That storage instance will then only be able to successfully store or retrieve headers corresponding to the correct ChainID. In addition, the height-based index will also be specific to that ChainID.

Currently, an exception is made for cluster chains: Since the ChainID changes when a new epoch begins, but collection nodes still access collections from the previous epoch/chain (to deduplicate transactions), storage instances for cluster chains are allowed to retrieve (but not store) headers from other cluster chains.
This should likely be further refactored.

Closes: #4204

Weakens the chainID requirement for cluster chains when reading from storage.

github-actions · 2025-12-04T00:08:53Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

codecov-commenter · 2025-12-04T00:12:03Z

Codecov Report

❌ Patch coverage is 9.75610% with 185 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...llection/epochmgr/mock/epoch_components_factory.go	0.00%	20 Missing ⚠️
state/protocol/badger/state.go	0.00%	20 Missing ⚠️
cmd/scaffold.go	0.00%	16 Missing ⚠️
storage/store/headers.go	37.50%	6 Missing and 4 partials ⚠️
state/protocol/util/testing.go	0.00%	9 Missing ⚠️
...ine/collection/epochmgr/factories/cluster_state.go	0.00%	8 Missing ⚠️
...ck-executed-height/cmd/rollback_executed_height.go	0.00%	5 Missing ⚠️
cmd/util/cmd/verify-evm-offchain-replay/verify.go	0.00%	5 Missing ⚠️
...d/util/cmd/exec-data-json-export/block_exporter.go	0.00%	4 Missing ⚠️
...d/exec-data-json-export/delta_snapshot_exporter.go	0.00%	4 Missing ⚠️
... and 27 more

📢 Thoughts on this report? Let us know!

durkmurder · 2025-12-09T14:55:54Z

storage/store/headers.go

+		}
+		// raise an error when the retrieved header is for a different chain than expected,
+		// except in the case of cluster chains where the previous epoch(=chain) can be checked for transaction deduplication
+		if header.ChainID != chainID && !(isClusterChain(chainID) && isClusterChain(header.ChainID)) {


I am not particularly fond of this. Seems like a workaround for a particular case, additionally we rely on the naming of the chain ID to be specific for recognizing a cluster. Additionally, I don't see that sentinel error is explained in the general interface.

I would propose next:

Update the API to describe newly added sentinel error.

Avoid any specific workarounds for specific chain IDs and just return a sentinel if there is a wrong chain. The user of the interface needs to deal with this on his own but not rely on the implementation detail of the storage layer.

+1 - See my comment here https://github.com/onflow/flow-go/pull/8222/files#r2608027971, I think we can remove the problematic part of this.

jordanschalm · 2025-12-10T19:30:06Z

cmd/util/cmd/read-badger/cmd/collections.go

+			if err != nil {
+				return err
+			}
+			storages := common.InitStorages(db, chainID) // TODO(4204) - header storage not used


Did you want to address this TODO in this PR?

jordanschalm · 2025-12-10T19:31:30Z

cmd/scaffold.go

 }

+func (fnb *FlowNodeBuilder) determineChainID() error {
+	if ok, _ := badgerState.IsBootstrapped(fnb.ProtocolDB); ok {


We should handle the error from IsBootstrapped

jordanschalm · 2025-12-10T19:32:20Z

state/protocol/badger/state.go

+
+// GetLatestFinalizedHeader attempts to retrieve the latest finalized header
+// without going through the storage.Headers interface.
+func GetLatestFinalizedHeader(db storage.DB) (*flow.Header, error) {


document expected errors here

jordanschalm · 2025-12-10T19:32:23Z

state/protocol/badger/state.go

 	return true, nil
 }

+func GetChainIDFromLatestFinalizedHeader(db storage.DB) (flow.ChainID, error) {


document expected errors here

jordanschalm · 2025-12-10T19:34:24Z

cmd/scaffold.go

+func (fnb *FlowNodeBuilder) determineChainID() error {
+	if ok, _ := badgerState.IsBootstrapped(fnb.ProtocolDB); ok {
+		chainID, err := badgerState.GetChainIDFromLatestFinalizedHeader(fnb.ProtocolDB)
+		if err == nil {


I would invert this to handle the err != nil case in the conditional (return unexpected error) and set the chain ID outside the conditional. Otherwise we are ignoring unexpected errors, which we should generally never do.

jordanschalm · 2025-12-10T19:37:16Z

engine/collection/epochmgr/factory.go

+	// ChainID refers to the consensus chain, from which reference blocks are used.
 	//
 	// Must return ErrNotAuthorizedForEpoch if this node is not authorized in the epoch.
-	Create(epoch protocol.CommittedEpoch) (
+	Create(epoch protocol.CommittedEpoch, chainID flow.ChainID) (


Suggested change

// ChainID refers to the consensus chain, from which reference blocks are used.

//

// Must return ErrNotAuthorizedForEpoch if this node is not authorized in the epoch.

Create(epoch protocol.CommittedEpoch) (

Create(epoch protocol.CommittedEpoch, chainID flow.ChainID) (

//

// Must return ErrNotAuthorizedForEpoch if this node is not authorized in the epoch.

Create(epoch protocol.CommittedEpoch, consensusChainID flow.ChainID) (

jordanschalm · 2025-12-10T19:55:17Z

module/builder/collection/builder.go


 	for _, blockID := range clusterBlockIDs {
-		header, err := b.clusterHeaders.ByBlockID(blockID)
+		header, err := b.clusterHeaders.ByBlockID(blockID) // TODO(4204) transaction deduplication crosses clusterHeaders epoch boundary


Transaction de-duplication actually does not occur across cluster and epoch boundaries.

Each transaction is uniquely assigned to one cluster in one epoch, based on the transaction's reference block (see ingestion logic)

Therefore, each cluster has a range of reference block heights they can accept. These ranges are equivalent to the height range of blocks within an epoch ($[FirstBlockInEpoch.Height, LastBlockInEpoch.Height]$. These ranges are consecutive and do not overlap.

In short, if we are considering a cluster block with reference block height $FirstBlockInEpoch.Height$, then minRefHeight is actually $FirstBlockInEpoch.Height$ (we don't need to search further back).

We already take this into account when determining the lowest possible reference block

So I think we can remove this TODO, and remove the special-case logic in storage.Headers meant to work around this. I would also suggest adding some documentation here explaining why there is no overlap between clusters and epochs.

jordanschalm · 2025-12-10T19:56:26Z

storage/store/headers.go

+		}
+		// raise an error when the retrieved header is for a different chain than expected,
+		// except in the case of cluster chains where the previous epoch(=chain) can be checked for transaction deduplication
+		if header.ChainID != chainID && !(isClusterChain(chainID) && isClusterChain(header.ChainID)) {


+1 - See my comment here https://github.com/onflow/flow-go/pull/8222/files#r2608027971, I think we can remove the problematic part of this.

jordanschalm · 2025-12-10T20:00:27Z

storage/store/headers.go

 		return operation.InsertHeader(lctx, rw, blockID, header)
 	}

+	isClusterChain := func(chainID flow.ChainID) bool {


I would suggest defining this as a method on flow.ChainID to consolidate the logic and documentation. I also think we can improve safety slightly:

Create a new constructor NewClusterHeaders

NewClusterHeaders returns an error if if thinks the chain ID input is not a cluster chain

NewHeaders returns an error if if thinks the chain ID input is a cluster chain

Each constructor binds the appropriate height lookup function for the kind of header it is for. (Most of the constructor logic can go into a shared, private newHeaders function that accepts the height lookup as an argument).

I prefer this because it makes the clients expectations more explicit. If the IsClusterChain logic fails to match the constructor used, then we will error, rather than continuing in an inconsistent state.

tim-barry added 8 commits December 1, 2025 14:53

add ChainID parameter to Header storage

f3c7603

update cluster mutator/snapshot tests

0fe4007

update header generation in cluster builder tests

467b732

update mock usage in epochmgr tests

585c0e9

enable TestExtend_WithReferenceBlockFromClusterChain

6d9c389

fix FinalizedAncestryLookup during cluster switchover

927b229

Weakens the chainID requirement for cluster chains when reading from storage.

Use appropriate height index for header storage

81ffdeb

Merge branch 'master' into tim/4204-split-header-storage-by-chainid

99a2498

tim-barry requested review from durkmurder and jordanschalm December 4, 2025 00:08

tim-barry and others added 4 commits December 4, 2025 11:45

Merge branch 'master' into tim/4204-split-header-storage-by-chainid

6253765

introduce sentinel error for incorrect header chain

a22fd0b

update default ChainID for cluster block fixture in tests

ee80525

update tests

2ecfee1

tim-barry marked this pull request as ready for review December 8, 2025 18:03

tim-barry requested a review from a team as a code owner December 8, 2025 18:03

durkmurder reviewed Dec 9, 2025

View reviewed changes

jordanschalm reviewed Dec 10, 2025

View reviewed changes

Differentiate between Consensus and Cluster Headers storage #8222

Are you sure you want to change the base?

Differentiate between Consensus and Cluster Headers storage #8222

Uh oh!

Conversation

tim-barry commented Dec 4, 2025

Uh oh!

github-actions bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

codecov-commenter commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Dec 4, 2025 •

edited

Loading

codecov-commenter commented Dec 4, 2025 •

edited

Loading