statement-store performance benchmarks

[AndreiEres]

Description

Adds benchmarks to measure the performance of the statement-store:

• Message Exchange Scenario: interaction with one or many nodes.
• Memory Stress Test Scenario.

Results

Key improvements made to improve the performance:

• Fixed a deadlock
• Increased statements limits
• Improved gossiping

Hardware
All benchmarks were run on a MacBook Pro M2

1. Message Exchange Scenario

Test Configuration

• Total participants: 49_998
• Group size: 6 participants per group
• Total groups: 8_333
• Statement payload size: 512 KB
• Propagation delay: 2 seconds (empirically determined)
• Parachain network tested with: 2, 3, 6, 12 nodes

Network Topologies
We tested two distribution patterns:

1. To one RPC: All participants send statements to a single RPC node
2. To all RPC: Participants distribute statements across all nodes (slower due to gossiping overhead)

Participant Flow

• Sends a statement with their key for an exchange session (1 sent)
• Waits 2 seconds for statement propagation
• Receives session keys from other members in the group (5 received)
• Sends statements containing a 512KB message to each member in the group (5 sent)
• Waits 2 seconds for statement propagation
• Receives messages from other members (5 received)
• Total: 6 sent, 10 received.

Results

Collators	Avg time	Max time	Memory
To one RPC
2	35s	35s	2.1GB
6	48s	50s	1.7GB
To all RPC
3	41s	51s	1.9GB
6	61s	71s	1.4GB
12	94s	119s	1.9GB

Observations

• Sending to one RPC node is faster but creates a bottleneck
• Distributing across all nodes takes longer due to gossiping overhead
• More collators increase gossiping load, resulting in slower completion times
• Memory usage per node remains around 2GB.

2. Memory Stress Test Scenario

Test Configuration
We prepared one more scenario to check how much memory nodes use with a full store after we increased the limits. To maximize memory usage for index usage, we submitted statements with all unique topics; other fields (e.g., proofs) were not used.

• Total tasks: 100,000 concurrent
• Statement size: 1 KB per statement
• Network size: 6 collators

Test Flow

1. Spawn 100,000 concurrent tasks
2. Each task sends statements with 1KB payload to one node until store is full
3. Statements are gossiped across the network to the other 5 collators
4. Test completes when all collator stores are full

Results
During the tests, each node used up to 4.5GB of memory.

[paritytech-workflow-stopper]

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/18411164414
Failed job name: fmt

[alexggh]

The tests are fine, but if gossiping is the bottleneck we should also have some results where we don't run all the nodes on the same machine.

[alexggh]

Why is this needed ?

[alexggh]

I do think that for a high number of collators, the performance starts to get negatively impacted by the fact that you are running on the same machine so many node.

Another thing that probably impacts things is that gossip times will be probably smaller when you have nodes running on the same machine.

[alexggh]

Why can't we use an existing spec how is this different ?

[gui1117]

if this is the binary George sent, I think we can just modify people-rococo to just have validate-statement implementation as needed for the benchmark, anyway rococo is not live AFAIK.

[alexggh]

Got it, so we need it because it is a rococo with statement-store pallet, let's just document the somewhere, so we don't forget what is in there.

[alexggh]

Let's add some documentation to each of the tests what scenario is running and what is outputting.