feat(gossipsub): introduce backpressure #5595

jxs · 2024-09-10T22:34:39Z

Description

superseeds #4914 with some changes and improvements, namely:

introduce a Delay for Forward and Publish messages, messages that take more than the configured delay to be sent are discarded
introduce scoring and penalize slow peers
remove control pool
report slow peers with the number of failed messages

ackintosh

I’ve left a few small comments.

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/peer_score.rs

guillaumemichel

It would be great to add new tests for the introduced backpressure

protocols/gossipsub/src/peer_score/params.rs

protocols/gossipsub/src/config.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/peer_score.rs

protocols/gossipsub/src/behaviour.rs

jxs

Thanks for the reviews Akihito and Gui!

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/peer_score.rs

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/config.rs

protocols/gossipsub/src/types.rs

…ipsub-backpressure-2

jxs

It would be great to add new tests for the introduced backpressure

yup for sure thanks for the suggestion Gui! Added some tests

ackintosh

Thanks João, LGTM!

elenaf9

Few comments and nit picks.

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/handler.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/config.rs

protocols/gossipsub/src/error.rs

protocols/gossipsub/src/handler.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/types.rs

mergify · 2024-11-21T12:48:07Z

This pull request has merge conflicts. Could you please resolve them @jxs? 🙏

refactor(gossipsub): use `send_message` for all RpcOut

Co-authored-by: Elena Frank <[email protected]>

fix(gossipsub): readd lines accidently removed with #12

Co-authored-by: Elena Frank <[email protected]>

jxs

Think I addressed everything, can you take another look Elena? Thanks!

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/error.rs

protocols/gossipsub/src/config.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/handler.rs

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/handler.rs

…ipsub-backpressure-2

elenaf9 · 2024-11-22T22:08:30Z

I think the new rpc module was accidentally not included in 602c46f? 😇
But everything else looks good to me! Just one remaining open comment about whether in the case that the channel is full and publishing failed, we should also update the metrics same as when a message is dropped?

jxs · 2024-11-23T12:47:01Z

I think the new rpc module was accidentally not included in 602c46f? 😇 But everything else looks good to me! Just one remaining open comment about whether in the case that the channel is full and publishing failed, we should also update the metrics same as when a message is dropped?

ah right! Added the rpc module. The unresolved conversations were from before your PR was merged, now metrics are being record on send_message and cover the cases you commented right?

elenaf9

LGTM!

jxs force-pushed the impl-gossipsub-backpressure-2 branch from 6d406e1 to 5e349f5 Compare September 10, 2024 22:44

feat(gossipsub): implement backpressures

00cde64

jxs force-pushed the impl-gossipsub-backpressure-2 branch from 5e349f5 to 00cde64 Compare September 10, 2024 22:52

ackintosh reviewed Sep 15, 2024

View reviewed changes

protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved

protocols/gossipsub/src/peer_score.rs Outdated Show resolved Hide resolved

guillaumemichel self-requested a review September 24, 2024 14:49

guillaumemichel reviewed Sep 26, 2024

View reviewed changes

address review

cddab5a

jxs commented Oct 2, 2024

View reviewed changes

jxs mentioned this pull request Oct 25, 2024

feat(gossipsub): apply max_transmit_size to the published message #5642

Merged

4 tasks

jxs added 2 commits October 31, 2024 10:26

address Gui review

e7b3b92

Merge branch 'master' of github.com:libp2p/rust-libp2p into impl-goss…

887fe07

…ipsub-backpressure-2

jxs force-pushed the impl-gossipsub-backpressure-2 branch from 8564d5e to 887fe07 Compare October 31, 2024 10:27

jxs commented Oct 31, 2024

View reviewed changes

jxs force-pushed the impl-gossipsub-backpressure-2 branch from ee51578 to ec0e0b4 Compare November 4, 2024 10:29

jxs added 2 commits November 4, 2024 11:14

count failed publish and forwards when not send from behaviour

7c7185c

add tests

c9e6602

jxs force-pushed the impl-gossipsub-backpressure-2 branch from ec0e0b4 to c9e6602 Compare November 4, 2024 12:07

jxs requested review from ackintosh and guillaumemichel November 4, 2024 14:58

ackintosh approved these changes Nov 4, 2024

View reviewed changes

elenaf9 reviewed Nov 18, 2024

View reviewed changes

elenaf9 reviewed Nov 19, 2024

View reviewed changes

guillaumemichel approved these changes Nov 19, 2024

View reviewed changes

refactor(gossipsub): use send_message for all RpcOut

6a3b59e

elenaf9 mentioned this pull request Nov 20, 2024

refactor(gossipsub): use send_message for all RpcOut jxs/rust-libp2p#12

Merged

4 tasks

elenaf9 reviewed Nov 20, 2024

View reviewed changes

protocols/gossipsub/src/behaviour.rs Show resolved Hide resolved

protocols/gossipsub/src/types.rs Show resolved Hide resolved

elenaf9 added 2 commits November 21, 2024 10:50

refactor(gossipsub): return RpcOut in RpcSender::send_message Err

455edd6

fix(gossipsub): make clippy happy

de1bf9f

Merge pull request #12 from elenaf9/gossipsub/sender_send_message

b626118

refactor(gossipsub): use `send_message` for all RpcOut

fix(gossipsub): readd lines accidently removed with #12

b295f56

elenaf9 mentioned this pull request Nov 22, 2024

fix(gossipsub): readd lines accidently removed with jxs#12 jxs/rust-libp2p#13

Merged

4 tasks

jxs and others added 5 commits November 22, 2024 15:43

Update protocols/gossipsub/src/behaviour.rs

c812627

Co-authored-by: Elena Frank <[email protected]>

Merge pull request #13 from elenaf9/gossipsub/backpressure/readd-logs

1789386

fix(gossipsub): readd lines accidently removed with #12

Apply suggestions from code review

7665b2d

Co-authored-by: Elena Frank <[email protected]>

Update protocols/gossipsub/src/types.rs

d795118

Co-authored-by: Elena Frank <[email protected]>

Apply suggestions from code review

6fc1bb2

Co-authored-by: Elena Frank <[email protected]>

jxs commented Nov 22, 2024

View reviewed changes

jxs added 2 commits November 22, 2024 19:04

seprate RpcSender and RpcReceiver to their own modules

602c46f

Merge branch 'master' of github.com:libp2p/rust-libp2p into impl-goss…

37de20c

…ipsub-backpressure-2

add missing rpc module

658ab16

elenaf9 approved these changes Nov 25, 2024

View reviewed changes

jxs force-pushed the impl-gossipsub-backpressure-2 branch from b60cfd6 to e1c06db Compare November 25, 2024 11:51

jxs added 3 commits November 25, 2024 12:28

replace and fix timeout messages with a field

2789c42

restore priority_cap

0ec96fa

add dots to peer_score comments

d8e3957

jxs force-pushed the impl-gossipsub-backpressure-2 branch from f772792 to d8e3957 Compare November 25, 2024 12:30

jxs added the send-it label Nov 25, 2024

mergify bot merged commit 2371922 into libp2p:master Nov 25, 2024
70 of 71 checks passed

drHuangMHT mentioned this pull request Nov 29, 2024

feat(gossipsub): introduce backpressure #4914

Closed

4 tasks

drHuangMHT mentioned this pull request Dec 10, 2024

Backpressure in Gossipsub #4667

Closed

jxs mentioned this pull request Dec 10, 2024

Should a NetworkBehavior be informed about a dropped ToSwarm::NotifyHandler event? #5718

Open

erhant mentioned this pull request Dec 30, 2024

bug: resource usage firstbatchxyz/dkn-compute-node#140

Closed

jxs mentioned this pull request Jan 11, 2025

fix(gossipsub): send iwant messages immediately #4875

Closed

4 tasks

jxs mentioned this pull request Mar 3, 2025

Delayed RPC Send Using Tokens sigp/lighthouse#5923

Merged

jxs mentioned this pull request Aug 9, 2025

Gossipsub backpressure seems to not work for forwarded messages #6117

Open

feat(gossipsub): introduce backpressure #5595

feat(gossipsub): introduce backpressure #5595

Conversation

jxs commented Sep 10, 2024

Description

Uh oh!

ackintosh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

guillaumemichel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jxs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jxs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ackintosh left a comment

Choose a reason for hiding this comment

Uh oh!

elenaf9 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Nov 21, 2024

Uh oh!

jxs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elenaf9 commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jxs left a comment •

edited

Loading

elenaf9 commented Nov 22, 2024 •

edited

Loading