async commit + wh

[nischitpra]

Summary by CodeRabbit

• New Features

  • Added support for a read-only ClickHouse instance for API queries, enabling read/write separation. Configurable via environment variables or YAML, with automatic fallback to the main instance.

• Documentation

  • New guide for setting up and operating a read-only ClickHouse, plus an example configuration file and troubleshooting tips.

• Refactor

  • Centralized storage initialization and introduced concurrent publishing/committing in the orchestrator to improve performance and scalability.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

Adds environment-driven readonly ClickHouse config, a readonly storage connector, and updates handler initialization to select readonly or main storage. Introduces docs and example config. Refactors committer to concurrently publish and commit per batch. Adds a generic NewReadonlyConnector factory and removes an older handler-based storage singleton.

Changes

Cohort / File(s)	Summary
Config: Readonly ClickHouse settings configs/config.go	Adds ClickhouseConfig fields: ReadonlyHost, ReadonlyPort, ReadonlyUsername, ReadonlyPassword, ReadonlyDatabase; reads corresponding env vars in setCustomJSONConfigs.
Example configuration configs/config_readonly_example.yml	Adds example YAML showing main and readonly ClickHouse settings, env var names, and related tuning options.
Documentation docs/README_READONLY_CLICKHOUSE.md	New doc explaining readonly ClickHouse usage, configuration, affected endpoints, setup, and operations.
Handlers: storage accessor refactor internal/handlers/logs_handlers.go, internal/handlers/storage_utils.go	Removes old lazy singleton from logs_handlers; adds storage_utils with singleton getMainStorage choosing readonly vs main connector based on config.
Orchestrator: commit/publish flow internal/orchestrator/committer.go	Removes “parallel mode”; per batch, concurrently publishes to Kafka and commits to storage; decouples cleanup from publish; adjusts metrics updates.
Storage: readonly ClickHouse connector internal/storage/clickhouse_readonly.go	Adds ClickHouseReadonlyConnector with readonly connection setup, read APIs, guarded panics for write operations, and simple query builder/helpers.
Storage: connector factory (readonly) internal/storage/connector.go	Adds NewReadonlyConnector[T] selecting Postgres (placeholder) or ClickHouse readonly connector; returns typed connector or error.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant API Handler
  participant StorageUtils as storage_utils.getMainStorage
  participant Factory as storage.New{Readonly,}Connector
  participant CH_RO as ClickHouse Readonly
  participant CH_Main as ClickHouse Main

  Client->>API Handler: HTTP request
  API Handler->>StorageUtils: getMainStorage()
  StorageUtils->>StorageUtils: Check Clickhouse.Readonly* config
  alt Readonly configured
    StorageUtils->>Factory: NewReadonlyConnector(cfg)
    Factory->>CH_RO: Connect (readonly)
    Factory-->>StorageUtils: readonly storage
  else Fallback
    StorageUtils->>Factory: NewConnector(cfg)
    Factory->>CH_Main: Connect (main)
    Factory-->>StorageUtils: main storage
  end
  API Handler->>StorageUtils: Use storage (reads)
  StorageUtils-->>API Handler: Query results
  API Handler-->>Client: Response

Loading

sequenceDiagram
  participant Orchestrator
  participant Committer
  participant Kafka
  participant Storage

  Orchestrator->>Committer: Start
  loop runCommitLoop (per batch)
    Committer->>Committer: Fetch blockDataToCommit
    par Publish
      Committer->>Kafka: Publish blockDataToCommit
      Committer->>Committer: Update lastPublishedBlock
    and Commit
      Committer->>Storage: Insert/commit blocks
      Committer->>Committer: Update metrics
      Committer->>Committer: cleanupProcessedStagingBlocks (async)
    end
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 13db8af and acd4465.

📒 Files selected for processing (8)

• configs/config.go (3 hunks)
• configs/config_readonly_example.yml (1 hunks)
• docs/README_READONLY_CLICKHOUSE.md (1 hunks)
• internal/handlers/logs_handlers.go (0 hunks)
• internal/handlers/storage_utils.go (1 hunks)
• internal/orchestrator/committer.go (1 hunks)
• internal/storage/clickhouse_readonly.go (1 hunks)
• internal/storage/connector.go (1 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch np/read_only_clickhouse

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.