feat(hermes): add staleness histograms for publish→receive and receive→broadcast latencies

[devin-ai-integration]

Summary

Adds two new histogram metrics to track staleness/latency in the Hermes price data pipeline:

• publish_to_receive_latency_seconds: Measures time from message publish_time to Hermes receive_time (ingestion latency)
• sse_broadcast_latency_seconds/ws_broadcast_latency_seconds: Measures time from Hermes receive_time to broadcast over SSE/WebSocket (broadcast latency)

Rationale

These metrics help identify bottlenecks and pressure points in Hermes that may be causing OOM issues or slow broadcasts. The first metric tracks how quickly Hermes ingests price data, while the second tracks broadcast delays that could indicate system pressure.

Key implementation decisions:

• Used aggregate-only histograms (no per-feed labels) to avoid high-cardinality metrics
• Applied seconds-based buckets (0.001-20s) consistent with existing latency metrics
• Placed observations at critical pipeline junctions: after message state creation and before broadcast

How has this been tested?

• Current tests cover my changes (existing test suite passes)
• Added new tests
• Manually tested the code (cargo build succeeds)

Testing performed:

• Verified Hermes compiles successfully with new metrics
• Confirmed metric registration patterns follow existing conventions
• Validated histogram buckets align with other latency metrics

Changes Made

1. Aggregate metrics (src/state/aggregate/metrics.rs):

   • Added publish_to_receive_latency histogram
   • Added observe_publish_to_receive() method

2. Ingestion observation (src/state/aggregate.rs):

   • Observe publish→receive latency after build_message_states()

3. API metrics (src/api/metrics_middleware.rs):

   • Added sse_broadcast_latency histogram for SSE broadcasts

4. SSE observation (src/api/rest/v2/sse.rs):

   • Observe receive→broadcast latency before sending price updates

5. WebSocket metrics (src/api/ws.rs):

   • Added broadcast_latency histogram
   • Observe receive→broadcast latency before sending messages

6. Version bump: apps/hermes/server/Cargo.toml 0.10.4 → 0.10.5

Review Checklist

⚠️ Critical items for review:

• Timestamp unit consistency: Verify publish_time, received_at, and current time are all in seconds (casts to f64)
• Missing timestamp handling: Check behavior when received_at or proof_available_time is None
• Performance impact: Assess overhead of metric observations in hot broadcast paths
• Field alignment: Confirm proof_available_time in SSE path corresponds to received_at conceptually
• Async registration pattern: Review histogram cloning approach for metric registration in async tasks

Additional considerations:

• Metric naming follows Prometheus conventions
• Histogram buckets appropriate for expected latency ranges
• No per-feed labels to avoid cardinality explosion

---

Link to Devin run: https://app.devin.ai/sessions/ac042ae7c2494cfd92e5d2e7c864274f
Requested by: Tejas Badadare ([email protected])

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[linear]

PF-396 Hermes price staleness metrics

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
api-reference	Ready	Preview	Comment	Aug 20, 2025 11:27pm
component-library	Ready	Preview	Comment	Aug 20, 2025 11:27pm
developer-hub	Ready	Preview	Comment	Aug 20, 2025 11:27pm
entropy-explorer	Ready	Preview	Comment	Aug 20, 2025 11:27pm
insights	Ready	Preview	Comment	Aug 20, 2025 11:27pm
proposals	Ready	Preview	Comment	Aug 20, 2025 11:27pm
staking	Ready	Preview	Comment	Aug 20, 2025 11:27pm

[devin-ai-integration]

Link to Devin run: https://app.devin.ai/sessions/ac042ae7c2494cfd92e5d2e7c864274f

Requested by: Tejas Badadare