Conversation
ma2bd
reviewed
Feb 14, 2026
linera-rpc/src/lib.rs
Outdated
| pub const KEY_PEM: &str = include_str!(concat!(env!("OUT_DIR"), "/private_key.pem")); | ||
|
|
||
| /// Maximum bit-shift exponent for u32: `1u32 << 31` is the largest valid shift before overflow. | ||
| const MAX_SHIFT_EXPONENT: u32 = 31; |
Contributor
There was a problem hiding this comment.
Please no hardcoded constants.
Contributor
There was a problem hiding this comment.
This one seems quite sensible. I can't imagine us changing 👀
ndr-ds
force-pushed
the
ndr-ds/replace_linear_retry_backoff_with_full_jitter_exponential_backoff
branch
2 times, most recently
from
2cbebb4 to
8244a6e
Compare
deuszx
reviewed
Feb 17, 2026
deuszx
approved these changes
Feb 17, 2026
afck
reviewed
Feb 17, 2026
linera-rpc/src/lib.rs
Outdated
| max_backoff: std::time::Duration, | ||
| ) -> std::time::Duration { | ||
| use rand::Rng as _; | ||
| let exponential_delay = base_delay.saturating_mul(1u32 << attempt.min(MAX_SHIFT_EXPONENT)); |
Contributor
There was a problem hiding this comment.
We could use 1u32.overflowing_shl(attempt).unwrap_or(u32::MAX). Then we don't need the constant.
Or even 2u32.saturating_pow(attempt)? I'm sure the compiler will realize it's a power of 2!
Contributor
Author
There was a problem hiding this comment.
I had claude create some test code and generate the assembly for it, and surprisingly the compiler (at least on edition 2021) doesn't optimize 2u32.saturating_pow(attempt) to a
shift, it generates a long branching sequence.
TIL though about 1u32.checked_shl(attempt).unwrap_or(u32::MAX), which literally does the same thing I was doing by hand ha and compiles to the
same 6-instruction sequence as the original shift, so we should be good.
ndr-ds
force-pushed
the
ndr-ds/replace_linear_retry_backoff_with_full_jitter_exponential_backoff
branch
from
8244a6e to
6ed001a
Compare
afck
approved these changes
Feb 17, 2026
ndr-ds
deleted the
ndr-ds/replace_linear_retry_backoff_with_full_jitter_exponential_backoff
branch
ndr-ds
added a commit
that referenced
this pull request
Feb 17, 2026
) The retry logic in gRPC client requests, gRPC subscription reconnection, and cross-chain message forwarding all used linear backoff without jitter (`delay * retry_count`). This creates a [thundering herd](https://en.wikipedia.org/wiki/Thundering_herd_problem) risk: when a validator goes down and comes back up, all clients that were retrying wake up at nearly the same time and hit the recovering validator simultaneously, potentially bringing it down again. This happens because linear backoff is deterministic — every client on the same retry count sleeps the exact same duration, so their retries synchronize into bursts. Replace all three retry sites with [Full Jitter exponential backoff](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/), the industry-standard approach recommended by [AWS](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html), [Google Cloud](https://cloud.google.com/storage/docs/retry-rategy), and the [gRPC spec](https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md). The formula is: `sleep = random(0, min(cap, base * 2^attempt))`. - Exponential growth spaces retries further apart over time - Full randomization (jitter across the entire range, not just an additive offset) desynchronizes clients so they spread their retries evenly instead of clustering - A fixed 30s cap prevents excessive wait times (Google Cloud uses 30s, AWS uses 20s) - CI - These changes should be backported to the latest `testnet` branch, then - be released in a validator hotfix.
ndr-ds
added a commit
that referenced
this pull request
Feb 17, 2026
) The retry logic in gRPC client requests, gRPC subscription reconnection, and cross-chain message forwarding all used linear backoff without jitter (`delay * retry_count`). This creates a [thundering herd](https://en.wikipedia.org/wiki/Thundering_herd_problem) risk: when a validator goes down and comes back up, all clients that were retrying wake up at nearly the same time and hit the recovering validator simultaneously, potentially bringing it down again. This happens because linear backoff is deterministic — every client on the same retry count sleeps the exact same duration, so their retries synchronize into bursts. Replace all three retry sites with [Full Jitter exponential backoff](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/), the industry-standard approach recommended by [AWS](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html), [Google Cloud](https://cloud.google.com/storage/docs/retry-rategy), and the [gRPC spec](https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md). The formula is: `sleep = random(0, min(cap, base * 2^attempt))`. - Exponential growth spaces retries further apart over time - Full randomization (jitter across the entire range, not just an additive offset) desynchronizes clients so they spread their retries evenly instead of clustering - A fixed 30s cap prevents excessive wait times (Google Cloud uses 30s, AWS uses 20s) - CI - These changes should be backported to the latest `testnet` branch, then - be released in a validator hotfix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The retry logic in gRPC client requests, gRPC subscription reconnection, and cross-chain message forwarding all used
linear backoff without jitter (
delay * retry_count). This creates a thundering herd risk: when a validator goes down and comes back up, all clients that were retrying wake up at nearly the same time and hit the recovering validator simultaneously, potentially bringing it down again.This happens because linear backoff is deterministic — every client on the same retry count sleeps the exact same
duration, so their retries synchronize into bursts.
Proposal
Replace all three retry sites with Full Jitter exponential backoff, the industry-standard approach recommended by AWS, Google Cloud, and the gRPC spec.
The formula is:
sleep = random(0, min(cap, base * 2^attempt)).spread their retries evenly instead of clustering
Test Plan
Release Plan
testnetbranch, then