feat(sync): pipelined concurrent backfill (3 → 51 blocks/sec, 17x) - #33
Merged
Conversation
Original backfill loop awaited ingest_one() per block sequentially. With ~300ms per block (REST /chain/blocks/<n> + eth_getLogs + DB write), peak throughput was 3 blocks/sec → 5.7 days to backfill 1.5M-block gap. Refactor to pipelined fetch: - futures::stream over [start..=cap] - buffered(N) — N concurrent fetches in flight - writes still serial in height order so the cursor never lands ahead of the data (sync invariant 2) - INDEXER_BACKFILL_CONCURRENCY env var (default 50, max 500) Measured on mainnet 2026-05-14: - concurrency=50 → 30 blocks/sec - concurrency=200 → 51 blocks/sec (write-bound from here) - ETA for 1.5M-block backfill drops from 138h to ~8h Future iteration: batch the writes (insert N blocks per tx) to lift the write ceiling further. Out of scope for this PR.
|
Caution Review failedPull request was closed or merged during review No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR refactors the block backfill from a sequential height-by-height loop to a pipelined design: it concurrently fetches up to N block heights (configurable via INDEXER_BACKFILL_CONCURRENCY) while applying writes serially to preserve ordering. A new fetch_one helper performs fetch+conversion and returns Ok(None) for missing blocks; run_backfill logs progress, supports mid-pipeline cancellation (returning the current cursor), and advances the cursor on missing blocks. The PR also adds a repository-wide .github/CODEOWNERS entry. Sequence DiagramsequenceDiagram
participant run_backfill as run_backfill
participant fetch_queue as fetch_one queue
participant results as results buffer
participant write as write_block
participant storage as cursor/block storage
loop for each block height
run_backfill->>fetch_queue: spawn fetch_one (bounded concurrency)
end
loop process results in order
results->>run_backfill: Some(BlockBundle) or None
alt BlockBundle
run_backfill->>write: write_block(BlockBundle)
write->>storage: persist block
else None
run_backfill->>storage: write_cursor(height, 0) + warn log
end
run_backfill->>run_backfill: check cancellation
Note over run_backfill: periodic progress logging
end
run_backfill->>storage: return final cursor
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/sync/src/backfill.rs`:
- Around line 128-133: The code fails rustfmt: reformat the helper so it adheres
to cargo fmt style (run `cargo fmt --all`), e.g., adjust spacing and indentation
around the let bindings and match/else pattern; specifically reformat the block
containing retry_with_backoff, the let block_opt = ... line, the `let
Some(block) = block_opt else { return Ok(None); };` pattern, and the subsequent
provider.logs_in_range call (functions: retry_with_backoff, rest.block,
provider.logs_in_range, variables: block_opt, block) so the file compiles and
passes `cargo fmt -- --check`.
- Around line 70-76: Update the top-of-file/module header comments to describe
the current pipelined design: concurrent fetching of N blocks (tuned by
INDEXER_BACKFILL_CONCURRENCY and the backfill_concurrency() helper) with
sequential writes in height order so the cursor never advances past written
data; remove or change wording that says “sequential block-by-block ingest” and
that cancellation is only checked “between blocks” to instead state that
cancellation/checks occur around the fetch/write pipeline while preserving write
order. Ensure the header aligns with the inline comment and the
tracing::info!(concurrency, "backfill: pipelined fetch enabled") behavior.
- Around line 89-92: The loop in backfill.rs currently awaits
fetched.next().await before checking cancellation, which can block if an
in-flight fetch stalls; change the loop to use tokio::select! to race
cancel.cancelled() (or the cancellation future provided by your cancel token)
against fetched.next() so cancellation is observed immediately. Locate the loop
using fetched.next().await and the call to
cancel.is_cancelled()/cancel.cancelled(), replace the single await with a select
that returns early on the cancellation branch (logging and returning the cursor)
and otherwise processes the fetched (h, result) as before. Ensure you import
tokio::select! if not already available and preserve existing logging and return
semantics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 94cffcb7-9602-4d1c-89a1-6ad55b8d2b45
📒 Files selected for processing (2)
.github/CODEOWNERScrates/sync/src/backfill.rs
CR follow-up on PR #33: - cargo fmt: helper formatting - cancellation race: cancel issued during a slow fetch now returns within the cancel timeout; was waiting for in-flight item to land before checking. Use tokio::select! biased on cancel side.
5 tasks
This was referenced May 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Backfill was sequential
ingest_one()per block @ ~300ms each → 3 blocks/sec ceiling. With a 1.5M-block gap on mainnet (real situation 2026-05-14), ETA was 5.7 days.What
crates/sync/src/backfill.rs:futures::streamover[cursor+1..=cap]with.buffered(N)for N-concurrent fetchINDEXER_BACKFILL_CONCURRENCY(default 50, max 500) tunes the poolfetch_one()extracted to keepingest_one()(used by tail loop) untouchedMeasured
Live mainnet vps4 2026-05-14:
Future
Write path becomes the bottleneck once fetch is parallel. Next iteration: batch writes (insert N blocks per pg tx). Out of scope — separate PR.
Risk
Low. Pipeline preserves cursor-monotonicity. Cancellation honored mid-pipeline. 404 / damaged-block path unchanged from
ingest_one.Summary by CodeRabbit
New Features
Improvements
Chores