INFOPLAT-13349: feat(beholder): track bytes per export via custom gRPC stats handler

[kirqz23]

What

Adds a beholder.export.bytes metric to the beholder gRPC client, restoring
per-node log volume visibility (labelled by csa_public_key and signal_type logs/metrics) that was lost when
otelgrpc semconv v1.40.0 dropped rpc.client.request.size.

According to these docs, metrics are totally deprecated without any successors.
RPC semantic convention stability migration guide

Semantic conventions for RPC metrics

Changes

• Implements exportSizeHandler (gRPC stats.Handler) to capture outbound message sizes via OutPayload events
• Adds new metric beholder.export.bytes with attributes otel_signal and csa_public_key
• Wraps log and metric exporters with metering logic that:
  • Uses per-call context to isolate concurrent exports
  • Records bytes only on successful exports
  • Handles retries correctly (stores size, doesn't accumulate)
• Comprehensive test coverage including concurrency and retry scenarios

Requires

Supports

[github-actions]

👋 kirqz23, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-common

View full report

[Copilot]

Pull request overview

Adds a custom metric to restore per-node visibility into OTLP log export volume by capturing the uncompressed outbound gRPC payload size and emitting it as beholder.logs.export.bytes (labelled by csa_public_key).

Changes:

• Introduces a gRPC stats.Handler (sizeCaptureHandler) to capture stats.OutPayload.Length.
• Wraps the OTLP logs exporter with meteredLogsExporter to increment a bytes counter on successful exports (to avoid retry inflation).
• Wires the stats handler + metered exporter into NewGRPCClient’s shared log exporter connection.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
pkg/beholder/metered_exporter.go	Adds size-capture stats handler and metered exporter wrapper emitting beholder.logs.export.bytes.
pkg/beholder/client.go	Wires the shared capture + exporter wrapper and attaches the gRPC stats handler(s) to the log exporter dial options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kirqz23]

gRPC builds istats.NewCombinedHandler(...) from the registered slice of multiple handlers, that is exactly the delegating handler proposed by the Copilot. This proposal is exactly what gRPC already builds for us internally. The downside of having multiple stats handlers however is that gRPC fires every handler on every event, in registration order. So for one OutPayload event there will be two HandleRPC(...) calls which might be an overhead. We might consider either keeping both statsHandlers if we need them, or dropping grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts...)).

This old handler:

1. Emits rpc.client.* metrics (which part of them are going to be removed),
2. Creates a trace span per RPC
3. Injects trace-context headers

[kirqz23]

So, we have two approaches here:

1. Keep two stats handlers as we have it now. exportSizeHandler shouldn't add much to the performance.
2. Drop grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts...)) taking into account that we loose 3 points mentioned in the above comment, however rpc.client.* metrics are totally deprecated anyway expect duration, which can be added to our custom handler alongside beholder.export.bytes, e.g. sth like beholder.export.duration

cc @pkcll

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.