Optimize embedded db memory and disk

[jakeloo]

Summary by CodeRabbit

• New Features
  • Added experimental Pebble-backed storage for staging/orchestrator and a Pebble block buffer.
  • Introduced ClickHouse ingestion via a Null table with materialized views for blocks, transactions, logs, and traces.
• Changes
  • Simplified CLI/config: consolidated committer to a single to-block; removed several poller flags; adjusted S3 bucket description.
  • Badger is no longer selectable for main storage.
• Refactor
  • Improved poller concurrency/queuing and added richer worker logging.
  • Removed block-failure persistence across storage backends.
• Chores
  • Added new dependencies to support Pebble and related utilities.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

The change set replaces several poller and committer flags/configs, adds Pebble-based storage and buffering components, removes block-failure data structures and persistence from multiple backends, adjusts orchestrator control flow (validator creation, committer stop condition), reworks poller concurrency (queued/processing states with wait channels), introduces new ClickHouse ingest tables/materialized views, and updates dependencies.

Changes

Cohort / File(s)	Summary
CLI flags & bindings cmd/root.go	Removes multiple poller flags and bindings; retains some S3 and parallelism flags; replaces committer from/until with to-block; adds Pebble path flags for orchestrator/staging; updates migrator destination types; removes Badger migrator path.
Configuration structs configs/config.go	Shrinks PollerConfig (removes several fields); CommitterConfig renames UntilBlock→ToBlock; adds PebbleConfig and hooks it into orchestrator/staging; removes Badger from main storage; minor S3 comment cleanup.
Dependencies go.mod	Adds Pebble and related indirect deps; strictly additive.
Common types removal internal/common/block_failures.go	Deletes BlockFailure type and file.
Orchestrator - committer/validator internal/orchestrator/committer.go, internal/orchestrator/validator.go, internal/orchestrator/orchestrator.go	Commit boundary renamed to toBlock; cleanup now context-aware; Committer constructs Validator internally; adds Validator.EnsureValidBlocks; orchestrator no longer passes a Validator option.
Orchestrator - poller internal/orchestrator/poller.go	Replaces simple processing tracking with queued+processing states and per-range wait channels; adjusts Request/poll/lookahead flows; renames ErrBlocksProcessed→ErrBlocksProcessing; log/metric updates.
Worker logging internal/worker/worker.go	Adds first/last block fields to batch debug logs.
Storage - Pebble staging connector internal/storage/pebble.go	New PebbleConnector with TTL GC, range cache, staging data CRUD, and per-chain metadata getters/setters; gob-encoded payloads.
Storage - Pebble block buffer internal/storage/block_buffer_pebble.go	New PebbleBlockBuffer implementing IBlockBuffer: add/flush/query/clear/stats/close with gob encoding and size/block thresholds.
Storage connectors wiring internal/storage/connector.go	Adds “pebble__experimental” for orchestrator/staging; removes Badger as main storage option and its auto fallback.
Storage - Badger internal/storage/badger.go, internal/storage/block_buffer_badger.go	Tunes Badger options and switches to ZSTD; removes block-failure CRUD from BadgerConnector; temp dir prefix change for block buffer.
Storage - ClickHouse internal/storage/clickhouse.go	Removes block-failure CRUD methods and helpers.
Storage - Postgres internal/storage/postgres.go, internal/storage/postgres_connector_test.go	Removes block-failure CRUD and corresponding test.
ClickHouse ingest schema internal/tools/clickhouse_opts/0010_clickhouse_inserts_null_table_v1.sql, .../0011_clickhouse_inserts_null_table_v1_mv.sql	Adds inserts_null_table (Null engine) and 4 MVs to blocks/transactions/logs/traces with ARRAY JOINs.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Orchestrator
  participant Committer
  participant Validator
  participant Poller
  participant Staging as Storage (Pebble)
  participant RPC

  Orchestrator->>Committer: NewCommitter(rpc, storage, poller)
  Note over Committer: Constructs Validator internally
  Committer->>Validator: NewValidator(rpc, storage, worker)
  Orchestrator->>Committer: Start(ctx)

  loop Commit Loop
    Committer->>Poller: Request blocks (range)
    alt Available in staging
      Poller-->>Committer: Blocks
    else Processing in-flight
      Poller-->>Committer: waitForRange then fetch
    else No new blocks
      Poller-->>Committer: nil
    end

    Committer->>Validator: EnsureValidBlocks(ctx, blocks)
    alt Some invalid
      Validator->>RPC: Re-fetch invalid block numbers
      RPC-->>Validator: Raw blocks
      Validator-->>Committer: Validated blocks (combined)
    else All valid
      Validator-->>Committer: Blocks
    end

    Committer->>Staging: Commit/publish logic
    Committer->>Staging: cleanupProcessedStagingBlocks(ctx)
    alt Reached toBlock
      Committer-->>Orchestrator: stop
    end
  end

Loading

sequenceDiagram
  autonumber
  participant Client as Poll Requester
  participant Poller
  participant Queue as queuedRanges
  participant Proc as processingRanges
  participant Staging as Storage (Pebble)

  Client->>Poller: Request(chain, start..end)
  alt Range fully staged
    Poller->>Staging: Fetch range
    Staging-->>Poller: Data
    Poller-->>Client: Data
  else Range being processed
    Poller->>Proc: waitForRange(key)
    Proc-->>Poller: notify on completion
    Poller->>Staging: Fetch range (if present)
    Poller-->>Client: Data or nil
  else Range not queued/processing
    Poller->>Queue: markQueued(key)
    Poller->>Poller: enqueue work
    Note over Poller: On worker start:<br/>unmarkQueued → markProcessing<br/>on finish: notify waiters
    Poller-->>Client: ErrBlocksProcessing (caller retries)
  end

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between fcca23a and 46f052b.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (19)

• cmd/root.go (7 hunks)
• configs/config.go (3 hunks)
• go.mod (7 hunks)
• internal/common/block_failures.go (0 hunks)
• internal/orchestrator/committer.go (11 hunks)
• internal/orchestrator/orchestrator.go (1 hunks)
• internal/orchestrator/poller.go (13 hunks)
• internal/orchestrator/validator.go (1 hunks)
• internal/storage/badger.go (1 hunks)
• internal/storage/block_buffer_badger.go (2 hunks)
• internal/storage/block_buffer_pebble.go (1 hunks)
• internal/storage/clickhouse.go (0 hunks)
• internal/storage/connector.go (2 hunks)
• internal/storage/pebble.go (1 hunks)
• internal/storage/postgres.go (0 hunks)
• internal/storage/postgres_connector_test.go (0 hunks)
• internal/tools/clickhouse_opts/0010_clickhouse_inserts_null_table_v1.sql (1 hunks)
• internal/tools/clickhouse_opts/0011_clickhouse_inserts_null_table_v1_mv.sql (1 hunks)
• internal/worker/worker.go (1 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch jl/pebble

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.