Native Histograms

[Elvis339]

Classic Prometheus histograms require pre-defined fixed bucket boundaries (e.g. [0.005, 0.01, 0.025, 0.05, ...]). This has fundamental drawbacks:

• Buycket boundaries must be guessed upfront, before you know your latency distribution
• Wrong buckets produce poor quantile resolution or wasted cardinality
• Every bucket is a separate time series, making high-cardinality histograms expensive

Native histograms solve this by using an exponential bucket schema where boundaries are computed automatically via 2^(i / 2^schema). Key properties:

• No pre-defined buckets the schema determines resolution, not you
• Only buckets with observations are stored, so sparse distributions are cheap
• The entire histogram is a single time series regardless of resolution
• Quantile accuracy improves because buckets self-adapt to where data actually lands

This means the bucket-tuning work originally scoped in #1736 becomes largely obsolete we don't need to figure out the right bucket distribution if we're not defining buckets at all.

Current State

The histograms as currently defined have several problems that need to be addressed regardless of native histogram support:

1. Broken naming
2. Redundant manual counters.
3. Shared bucket distribution - all three histograms share one bucket config that wasn't chosen with any specific latency profile in mind

Read more here #1736

Proposed Plan

1. Histogram metrics followup #1736
2. Coordination with SREs
3. Switch to native histograms - replace classic histograms

Point 3. will be broken down by PRs and more in-depth explanations.

[Elvis339]

I have one concern regarding backward compatibility. Native histograms are only carried in the protobuf wire format they are silently dropped by text-based parsers. This means any consumer of

firewood/ffi/src/metrics.rs

Line 82 in b11fc0e

pub fn setup_metrics_with_exporter(metrics_port: u16) -> Result<(), Box<dyn Error>> {

(metrics server) that uses text format would lose histogram data entirely after the switch, with no error or warning.

Before we proceed I want to understand our current consumers:

1. Who is scraping metrics server today? We know it's not AvalancheGo because integration with AvalancheGo is via Gatherer.
2. Is the experimental workflow in (fwdctl) using histogram metrics in any assertions? If so those would silently degrade.

The metrics server feels like it's mostly experimental right now but I don't want to make that assumption without checking.

If we support both, protobuf and text based we risk native histograms representing in text as le="+Inf" only — a single bucket with no resolution, which is a silently degraded view. Consumers querying classic bucket labels like le="0.1" would get no data, while +Inf queries would still work but be meaningless for quantile estimation.

cc @rkuris

[Elvis339]

Supporting both format would cause dropping native histograms which is not an option. We're going to introduce breaking change for current text-format consumers. Prometheus already supports content negotiation to request protobuf via the Accept header.

[demosdemon]

No one is currently using this feature. The gather mechanism is the primary and main way for firewood users to consume metrics. The prometheus gather by setup_metrics_with_exporter is a legacy created that existed before the gather mechanism.

[Elvis339]

We have decided to explore more generic interface rather than raw Prometheus/protobuf to avoid locking into specific observability stack while still supporting meaningful and standardized data distribution.