Skip to content

Commit b7f843c

Browse files
committed
Fix double-distributed query
Use local offsets table in BatchFinder CTE to avoid double-distributed query error (ClickHouse error 288). The BatchFinder queries access_logs_federated (distributed table), which executes on each shard. Each shard should read its local offsets table rather than querying offsets_federated, which would create a distributed query within a distributed query. Both tables are sharded by raftSessionID, so data is co-located and each shard has the offsets it needs locally. CommitOffset and GetOffset continue to use offsets_federated for proper cross-shard routing.
1 parent 5462c1f commit b7f843c

2 files changed

Lines changed: 4 additions & 1 deletion

File tree

pkg/clickhouse/constants.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ const (
1111
// TableAccessLogsFederated is the distributed table across all shards
1212
TableAccessLogsFederated = "access_logs_federated"
1313

14+
// TableOffsets is the local table tracking processing offsets per bucket (MergeTree)
15+
TableOffsets = "offsets"
16+
1417
// TableOffsetsFederated is the distributed table for offsets across all shards
1518
TableOffsetsFederated = "offsets_federated"
1619
)

pkg/logcourier/batchfinder.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ func (bf *BatchFinder) FindBatches(ctx context.Context) ([]LogBatch, error) {
123123
OR min_ts <= now() - INTERVAL ? SECOND
124124
ORDER BY min_ts ASC
125125
LIMIT ?
126-
`, bf.database, clickhouse.TableOffsetsFederated, bf.database, clickhouse.TableAccessLogsFederated)
126+
`, bf.database, clickhouse.TableOffsets, bf.database, clickhouse.TableAccessLogsFederated)
127127

128128
rows, err := bf.client.Query(ctx, query, bf.countThreshold, bf.timeThresholdSec, bf.maxBuckets)
129129
if err != nil {

0 commit comments

Comments
 (0)