Skip to content

Partition token_balances #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 17, 2025
Merged

Partition token_balances #291

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db48264
Partition token_balances
jakeloo Sep 13, 2025
144585d
Index block number
jakeloo Sep 15, 2025
7e4530f
Update schema
jakeloo Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion internal/tools/clickhouse/0008_clickhouse_create_token_balances.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,6 @@ CREATE TABLE IF NOT EXISTS token_balances
)
)
ENGINE = ReplacingMergeTree(insert_timestamp, is_deleted)
PARTITION BY chain_id
PARTITION BY (chain_id, toStartOfQuarter(block_timestamp))
Copy link

@coderabbitai coderabbitai bot Sep 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Partition change won’t apply to existing deployments; add an explicit migration plan.

This file uses CREATE TABLE IF NOT EXISTS, so clusters that already have token_balances (partitioned by chain_id) will not be altered. You’ll ship a schema drift where fresh installs get (chain_id, quarter) while existing ones stay on chain_id-only.

Recommended migration (outline):

  • Create a v2 table with the new partition key.
  • Pause or dual-write ingestion.
  • Backfill via INSERT … SELECT.
  • Atomically swap tables; then drop the old one.
  • Do this ON CLUSTER if applicable.

Example SQL (adapt to your deploy tooling and cluster topology):

-- 1) Create new table
CREATE TABLE token_balances_v2
(
  -- same columns, indexes, projections as current token_balances
) ENGINE = ReplacingMergeTree(insert_timestamp, is_deleted)
PARTITION BY (chain_id, toStartOfQuarter(block_timestamp))
ORDER BY (chain_id, owner_address, token_address, token_id, block_number, transaction_index, log_index, direction)
SETTINGS index_granularity = 8192,
         lightweight_mutation_projection_mode = 'rebuild',
         deduplicate_merge_projection_mode = 'rebuild',
         allow_part_offset_column_in_projections = 1;

-- 2) Backfill
INSERT INTO token_balances_v2 SELECT * FROM token_balances;

-- 3) Swap (atomic)
RENAME TABLE token_balances TO token_balances_old, token_balances_v2 TO token_balances;

-- 4) Resume writes; validate; then drop old
DROP TABLE token_balances_old;

If this file is intended only for fresh installs, add a separate numbered migration that performs the above for upgrades to avoid silent divergence.

🤖 Prompt for AI Agents 
internal/tools/clickhouse/0008_clickhouse_create_token_balances.sql around line
65: the CREATE TABLE IF NOT EXISTS change adds a new PARTITION BY (chain_id,
toStartOfQuarter(block_timestamp)) but will not alter existing deployments that
already have token_balances partitioned only by chain_id, causing schema drift;
add an explicit migration that creates a token_balances_v2 with the new
partition key (matching all columns, indexes, projections and engine settings),
pause or dual-write ingestion if needed, backfill data via INSERT … SELECT to
token_balances_v2, perform an atomic swap (RENAME TABLE token_balances TO
token_balances_old, token_balances_v2 TO token_balances) and then resume writes
and validate before dropping token_balances_old, and ensure the migration runs
ON CLUSTER where applicable or provide this as an upgrade-only numbered
migration separate from the fresh-install DDL.

ORDER BY (chain_id, owner_address, token_address, token_id, block_number, transaction_index, log_index, direction)
SETTINGS index_granularity = 8192, lightweight_mutation_projection_mode = 'rebuild', deduplicate_merge_projection_mode = 'rebuild', allow_part_offset_column_in_projections=1;