Limit the size of the statement for further gossiping #9965

AndreiEres · 2025-10-08T08:35:42Z

Description

Limits the size of statements that are further gossiped over the network to prevent skipping oversized messages. The limit is set to match the network protocol's max statement notification size (1 MB), accounting for 1-byte vector length overhead because statements are sent as Vec<Statement>.

Integration

Affected crates:

sc-statement-store: Now depends on sc-network-statement for size constants
sc-network-statement: MAX_STATEMENT_NOTIFICATION_SIZE is now public

For downstream users:

Statements larger than 1 MB will now be rejected earlier before the validation pipeline

…e_dev --bump patch'

AndreiEres · 2025-10-08T08:36:10Z

/cmd prdoc --audience node_dev --bump minor

…e_dev --bump minor'

georgepisaltu · 2025-10-15T13:57:09Z

substrate/client/network/statement/src/config.rs


 /// Maximum allowed size for a statement notification.
-pub(crate) const MAX_STATEMENT_SIZE: u64 = 256 * 1024;
+pub const MAX_STATEMENT_SIZE: u64 = 256 * 1024;


Will be 1MiB after #9912

georgepisaltu · 2025-10-15T13:58:08Z

substrate/client/statement-store/src/lib.rs

 pub const DEFAULT_MAX_TOTAL_SIZE: usize = 2 * 1024 * 1024 * 1024; // 2GiB
+/// The maximum size of a single statement in bytes.
+/// Accounts for the 1-byte vector length prefix when statements are gossiped as `Vec<Statement>`.
+pub const MAX_STATEMENT_SIZE: usize = sc_network_statement::config::MAX_STATEMENT_SIZE as usize - 1;


Maybe we can do it the other way around and allow this as is and increase the MAX_STATEMENT_NOTIFICATION_SIZE value to what it is +1.

It creates cyclic package dependency with the other way around.

I was thinking of just moving the -1 from here to +1 there but anyway it's not that important.

gui1117 · 2025-10-16T02:49:00Z

substrate/client/statement-store/src/lib.rs

-		if statement_len > validation.max_size as usize {
+		if statement_len > validation.max_size as usize ||
+			statement.encoded_size() > MAX_STATEMENT_SIZE
+		{


we could also directly reject in the implementation of fn submit.
Both way are good.

Good catch, we can skip expensive validation with the early return in submittion

gui1117 · 2025-10-16T02:54:40Z

substrate/client/statement-store/src/lib.rs

 pub const DEFAULT_MAX_TOTAL_SIZE: usize = 2 * 1024 * 1024 * 1024; // 2GiB
+/// The maximum size of a single statement in bytes.
+/// Accounts for the 1-byte vector length prefix when statements are gossiped as `Vec<Statement>`.
+pub const MAX_STATEMENT_SIZE: usize = sc_network_statement::config::MAX_STATEMENT_SIZE as usize - 1;


I don't know how to make this constraint more visible.

People should be aware that this client implementation only support such statement size.

But for now it is good and better than before.

…es/max-statement-size

paritytech-workflow-stopper · 2025-10-16T09:01:10Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/18555597226
Failed job name: test-linux-stable

…es/max-statement-size

@lexnv

# Description Fixes gossiping and scalability issues in the statement-store networking. 1. Reduced gossiping traffic by propagating only recent statements instead of all. 2. Added an early check for statements that the node already has to skip duplicate processing. 3. Added splitting of large statement batches to stay under MAX_STATEMENT_NOTIFICATION_SIZE; oversized individual statements are skipped. 4. MAX_STATEMENT_NOTIFICATION_SIZE was updated to the commonly used 1MB, which drastically improved the gossiping speed. 5. Notifications are sent asynchronously. I don't see much difference in performance, but according to @lexnv, it's better to do: #9296. 6. Added a 10s timeout to handle very slow or disconnected peers. ## Integration Internal optimizations to the gossip protocol. No downstream changes required. Related PR: #9965 ## Things to handle in further PRs - After this PR, nodes don't send all statements to new peers anymore, only the recent ones. - After restarting, the node doesn't re-gossip statements it wasn't gossiped. - Broadcasting notifications to all peers when the first peer is slow is limited. We could instead use a FuturesUnordered. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bastian Köcher <[email protected]>

…ement-size

@lexnv

# Description Fixes gossiping and scalability issues in the statement-store networking. 1. Reduced gossiping traffic by propagating only recent statements instead of all. 2. Added an early check for statements that the node already has to skip duplicate processing. 3. Added splitting of large statement batches to stay under MAX_STATEMENT_NOTIFICATION_SIZE; oversized individual statements are skipped. 4. MAX_STATEMENT_NOTIFICATION_SIZE was updated to the commonly used 1MB, which drastically improved the gossiping speed. 5. Notifications are sent asynchronously. I don't see much difference in performance, but according to @lexnv, it's better to do: #9296. 6. Added a 10s timeout to handle very slow or disconnected peers. ## Integration Internal optimizations to the gossip protocol. No downstream changes required. Related PR: #9965 ## Things to handle in further PRs - After this PR, nodes don't send all statements to new peers anymore, only the recent ones. - After restarting, the node doesn't re-gossip statements it wasn't gossiped. - Broadcasting notifications to all peers when the first peer is slow is limited. We could instead use a FuturesUnordered. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bastian Köcher <[email protected]>

paritytech-release-backport-bot · 2025-10-22T02:55:49Z

Created backport PR for stable2506:

[stable2506] Backport #9965 #10075 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-9965-to-stable2506
git worktree add --checkout .worktree/backport-9965-to-stable2506 backport-9965-to-stable2506
cd .worktree/backport-9965-to-stable2506
git reset --hard HEAD^
git cherry-pick -x de740e1892f47e91c356858324f6e3e4b0f1be60
git push --force-with-lease

paritytech-release-backport-bot · 2025-10-22T02:55:53Z

Created backport PR for stable2509:

[stable2509] Backport #9965 #10076 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-9965-to-stable2509
git worktree add --checkout .worktree/backport-9965-to-stable2509 backport-9965-to-stable2509
cd .worktree/backport-9965-to-stable2509
git reset --hard HEAD^
git cherry-pick -x de740e1892f47e91c356858324f6e3e4b0f1be60
git push --force-with-lease

@lexnv

Fixes gossiping and scalability issues in the statement-store networking. 1. Reduced gossiping traffic by propagating only recent statements instead of all. 2. Added an early check for statements that the node already has to skip duplicate processing. 3. Added splitting of large statement batches to stay under MAX_STATEMENT_NOTIFICATION_SIZE; oversized individual statements are skipped. 4. MAX_STATEMENT_NOTIFICATION_SIZE was updated to the commonly used 1MB, which drastically improved the gossiping speed. 5. Notifications are sent asynchronously. I don't see much difference in performance, but according to @lexnv, it's better to do: #9296. 6. Added a 10s timeout to handle very slow or disconnected peers. Internal optimizations to the gossip protocol. No downstream changes required. Related PR: #9965 - After this PR, nodes don't send all statements to new peers anymore, only the recent ones. - After restarting, the node doesn't re-gossip statements it wasn't gossiped. - Broadcasting notifications to all peers when the first peer is slow is limited. We could instead use a FuturesUnordered. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bastian Köcher <[email protected]> (cherry picked from commit b21cbb5)

paritytech-release-backport-bot · 2025-10-22T06:22:49Z

Created backport PR for unstable2507:

[unstable2507] Backport #9965 #10079 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-9965-to-unstable2507
git worktree add --checkout .worktree/backport-9965-to-unstable2507 backport-9965-to-unstable2507
cd .worktree/backport-9965-to-unstable2507
git reset --hard HEAD^
git cherry-pick -x de740e1892f47e91c356858324f6e3e4b0f1be60
git push --force-with-lease

AndreiEres and others added 25 commits October 2, 2025 12:59

Gossip only recent statements

f06dab8

Check if the store already has a statement before processing

eca57b4

Reduce notification size if too large

7b8aeb1

Update metrics

3e69999

Update from github-actions[bot] running command 'prdoc --audience nod…

9a5decc

…e_dev --bump patch'

Update PRDOC

baeba9f

Update semver

b0607bc

Address review comments

52fb0aa

Fix types

f08c2a8

Fix up

1e3e83b

Fix up

8cdb31a

Fix typo

a317299

Remove unused var

50ecef4

Update validation queue size

6b89e1c

Add mocked handler for tests

a705c00

Add dirty test

8703601

Add more tests

90c0799

Fix duplicate processing

bd3645e

Add more tests

310e05b

Add metrics for ignored statements

ed55be3

Remove unused tokio

514ad77

Fix tests

0adf169

Fix style

6daaee0

Fix skipping very large statements

0012b9e

Limit the size of the statement for further gossiping

dad3ecd

AndreiEres added the T0-node This PR/Issue is related to the topic “node”. label Oct 8, 2025

github-actions bot and others added 2 commits October 8, 2025 08:38

Update from github-actions[bot] running command 'prdoc --audience nod…

6407d0e

…e_dev --bump minor'

Update pr_doc

46f7462

AndreiEres requested a review from georgepisaltu October 8, 2025 08:56

georgepisaltu reviewed Oct 15, 2025

View reviewed changes

gui1117 approved these changes Oct 16, 2025

View reviewed changes

AndreiEres added 2 commits October 16, 2025 10:17

Merge branch 'master' into AndreiEres/fix-statement-store-gossiping

b8a1c2f

Merge branch 'AndreiEres/fix-statement-store-gossiping' into AndreiEr…

34ce2d4

…es/max-statement-size

AndreiEres changed the base branch from master to AndreiEres/fix-statement-store-gossiping October 16, 2025 08:44

Fix name

fbaa076

AndreiEres added 4 commits October 16, 2025 12:04

Fix flaky test

a3d15d2

Merge branch 'AndreiEres/fix-statement-store-gossiping' into AndreiEr…

ac2cc30

…es/max-statement-size

Add early return

1ce8246

Update prdoc

633619f

Base automatically changed from AndreiEres/fix-statement-store-gossiping to master October 16, 2025 16:18

Merge remote-tracking branch 'origin/master' into AndreiEres/max-stat…

7f7b1d5

…ement-size

georgepisaltu approved these changes Oct 17, 2025

View reviewed changes

gui1117 added this pull request to the merge queue Oct 22, 2025

Merged via the queue into master with commit de740e1 Oct 22, 2025
246 checks passed

gui1117 deleted the AndreiEres/max-statement-size branch October 22, 2025 02:00

gui1117 added A4-backport-stable2506 Pull request must be backported to the stable2506 release branch A4-backport-stable2509 Pull request must be backported to the stable2509 release branch labels Oct 22, 2025

paritytech-release-backport-bot bot mentioned this pull request Oct 22, 2025

[stable2506] Backport #9965 #10075

Draft

paritytech-release-backport-bot bot mentioned this pull request Oct 22, 2025

[stable2509] Backport #9965 #10076

Open

gui1117 added the A4-backport-unstable2507 Pull request must be backported to the unstable2507 release branch label Oct 22, 2025

paritytech-release-backport-bot bot mentioned this pull request Oct 22, 2025

[unstable2507] Backport #9965 #10079

Draft

Limit the size of the statement for further gossiping #9965

Limit the size of the statement for further gossiping #9965

Uh oh!

Conversation

AndreiEres commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Integration

Uh oh!

AndreiEres commented Oct 8, 2025

Uh oh!

georgepisaltu Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

georgepisaltu Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

AndreiEres Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

georgepisaltu Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

gui1117 Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndreiEres Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

gui1117 Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

paritytech-workflow-stopper bot commented Oct 16, 2025

Uh oh!

Uh oh!

paritytech-release-backport-bot bot commented Oct 22, 2025

Uh oh!

paritytech-release-backport-bot bot commented Oct 22, 2025

Uh oh!

paritytech-release-backport-bot bot commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndreiEres commented Oct 8, 2025 •

edited

Loading

gui1117 Oct 16, 2025 •

edited

Loading