update clickhouse schemas to latest state #240
Conversation
WalkthroughSeveral ClickHouse SQL scripts were updated to enhance table definitions with new projections and table settings, optimizing data ordering and mutation handling. Additionally, a new Changes
Sequence Diagram(s)sequenceDiagram
participant LogsTable as logs
participant TokenTransfersMV as logs_to_token_transfers (MV)
participant TokenTransfersTable as token_transfers
LogsTable->>TokenTransfersMV: Insert new log entry
TokenTransfersMV->>TokenTransfersMV: Filter and parse log (ERC-20/721/1155)
TokenTransfersMV->>TokenTransfersMV: Extract addresses, token ids, amounts
TokenTransfersMV->>TokenTransfersTable: Insert parsed transfer event(s)
Estimated code review effort4 (~90 minutes) 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
internal/tools/clickhouse_create_logs_table.sql (1)
25-35: Solid projection but consider swappingtopic_0andaddressif address-centric queries dominateCurrent order:
(chain_id, topic_0, address, block_number, …).
If the most frequent pattern is “all events for contract X on chain Y”, puttingaddressbeforetopic_0will prune more quickly. Worth validating with real query stats; otherwise LGTM.internal/tools/clickhouse_create_token_transfers_mv.sql (1)
69-91: Repeated sub-query logic duplicates effort – extract CTE or reuse
transfer_logsis defined twice (before and afterUNION ALL). Duplicating the heavy logic doubles parse & compile time and makes maintenance error-prone.
Extract once and reuse, or perform the batch/regular split via a single SELECT with conditional array join.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
internal/tools/clickhouse_create_logs_table.sql(1 hunks)internal/tools/clickhouse_create_token_balances_mv.sql(1 hunks)internal/tools/clickhouse_create_token_transfers_mv.sql(1 hunks)internal/tools/clickhouse_create_transactions_table.sql(1 hunks)
🔇 Additional comments (6)
internal/tools/clickhouse_create_transactions_table.sql (2)
50-58: Nice: second projection already containshash– coverschain_id+to_address+hashlook-ups.
No action needed here.
61-62: Double-check new table-level settings on prod volumes
deduplicate_merge_projection_mode='drop'+lightweight_mutation_projection_mode='rebuild'can noticeably increase merge-tree write amplification and mutation cost on tens-of-TB partitions. Make sure this was load-tested (at least on staging) or gate-keep behind a feature flag.internal/tools/clickhouse_create_logs_table.sql (1)
38-40: Same caution on merge-tree settings as intransactionsVerify the write / compact performance with the new projection-aware mutation modes.
internal/tools/clickhouse_create_token_balances_mv.sql (1)
20-21: 👍 Explicitindex_granularity+ mutation mode – consistent with other tablesinternal/tools/clickhouse_create_token_transfers_mv.sql (2)
3-15: Type mismatch:amountcannot store negatives but you rely onsignonly
amountis declaredUInt256; the MV never stores negative values, relying onsignfor direction – that’s OK. Just confirm downstream consumers always multiply bysignwhen computing net flows.
16-45: Nice trio of projections – covers from / to / tx-hash predicates
Add ClickHouse Projections and Token Transfers Table
TL;DR
Added projections to existing ClickHouse tables and created a new token_transfers table with materialized view to improve query performance.
What changed?
logs_chainid_topic0_addressprojection to thelogstable to optimize queries filtering by chain_id, topic_0, and addressaddress_projectionto thetoken_balancestable to improve queries filtering by token_type, chain_id, and addresstoken_transferstable with three projections (from_address, to_address, transaction_hash) to efficiently track token transferslogs_to_token_transfersthat transforms logs into token transfer recordstransactionstable for queries filtering by chain_id with from_address or to_addressHow to test?
Why make this change?
ClickHouse projections provide pre-sorted data structures that significantly improve query performance for specific access patterns. The added projections optimize common query patterns like:
These changes will reduce query latency and resource consumption for these common access patterns.
Summary by CodeRabbit
New Features
Performance Improvements