Skip to content

feat(consumer): Emit batch_write_bytes metric from RowBinary writer#7863

Merged
onewland merged 1 commit intomasterfrom
oliver/batch-byte-size-metric
Apr 7, 2026
Merged

feat(consumer): Emit batch_write_bytes metric from RowBinary writer#7863
onewland merged 1 commit intomasterfrom
oliver/batch-byte-size-metric

Conversation

@onewland
Copy link
Copy Markdown
Contributor

@onewland onewland commented Apr 7, 2026

Summary

  • Adds the insertions.batch_write_bytes counter to the RowBinary ClickHouse writer, matching what the JSONEachRow writer already emits
  • This is Phase 0 of the byte-based batching rollout — we need to observe batch byte sizes on eap_items in s4s2/de (which use RowBinary) to pick a data-driven threshold for --max-batch-size-calculation=bytes

Refs EAP-460

Test plan

  • cargo check passes
  • Deploy to s4s2 and confirm insertions.batch_write_bytes appears in DataDog for RowBinary consumers
  • Use DataDog arithmetic (bytes/msgs) to approximate per-batch byte size distributions across all environments

🤖 Generated with Claude Code

The RowBinary writer was missing the insertions.batch_write_bytes counter
that the JSONEachRow writer already emits. Adding it here so we can observe
batch byte sizes across all eap_items consumers (s4s2/de use RowBinary)
as part of Phase 0 of the byte-based batching rollout.

Refs EAP-460

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@onewland onewland requested a review from a team as a code owner April 7, 2026 22:06
@linear-code
Copy link
Copy Markdown

linear-code bot commented Apr 7, 2026

@onewland onewland marked this pull request as draft April 7, 2026 22:08
Comment on lines 127 to 133
}
}

counter!("insertions.batch_write_bytes", num_bytes as i64);
counter!("insertions.batch_write_msgs", batch_len as i64);
empty_batch.record_message_latency();
empty_batch.emit_item_type_metrics();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The insertions.batch_write_bytes metric for the RowBinary writer uses an in-memory size estimate (estimated_size()) instead of the actual serialized byte count, making it inconsistent with the JSONEachRow writer.
Severity: MEDIUM

Suggested Fix

The RowBinary writer should measure the size of the data after it has been serialized into the RowBinary format, not before. This can be achieved by getting the length of the serialized byte buffer, which will ensure the metric reflects the actual number of bytes written and is consistent with the JSONEachRow writer's implementation.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: rust_snuba/src/strategies/clickhouse/row_binary_writer.rs#L127-L133

Potential issue: The `insertions.batch_write_bytes` metric for the RowBinary writer is
calculated using `estimated_size()`, which measures the in-memory size of the data
structures. This is inconsistent with the JSONEachRow writer, which measures the actual
byte length of the serialized data. Because the RowBinary format uses variable-length
encoding, the in-memory size is not an accurate representation of the on-the-wire size.
This discrepancy will result in inaccurate and non-comparable metric data between the
two writers, which could lead to incorrect decisions about batching thresholds and
undermines the goal of standardizing this metric.

Did we get this right? 👍 / 👎 to inform future reviews.

@onewland onewland marked this pull request as ready for review April 7, 2026 22:20
@onewland onewland merged commit a5c76af into master Apr 7, 2026
46 of 47 checks passed
@onewland onewland deleted the oliver/batch-byte-size-metric branch April 7, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants