Skip to content

Commit c50c800

Browse files
zeitlingerCopilot
zeitlinger
and
Copilot
authored
Verify and document native histograms with custom buckets (NHCB) support (#1846)
Fixes #1838 ## Summary Addresses #1838 by verifying and documenting that `client_java` fully supports native histograms with custom buckets (NHCB, schema -53). ## Background According to the [Prometheus Native Histograms specification](https://prometheus.io/docs/specs/native_histograms/), NHCB is handled by: 1. Clients expose classic histograms with custom bucket boundaries 2. Prometheus servers convert them to native histograms (schema -53) when configured with `convert_classic_histograms_to_nhcb: true` 3. Custom bucket boundaries are preserved in the conversion ## Changes ### 1. Comprehensive Test Suite ✅ - **File:** `prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/CustomBucketsHistogramTest.java` - **Tests:** 11 comprehensive tests (all passing) - **Coverage:** - Arbitrary custom bucket boundaries - Linear boundaries (equal-width buckets) - Exponential boundaries (for data spanning multiple orders of magnitude) - Classic-only and dual-mode histograms - Text and protobuf format serialization - Labeled histograms with custom buckets - Boundary edge cases and fine-grained precision ### 2. Documentation ✅ - **File:** `docs/content/getting-started/metric-types.md` - **Added:** - Custom Bucket Boundaries section with examples for arbitrary, linear, and exponential boundaries - Native Histograms with Custom Buckets (NHCB) section explaining: - How NHCB works - Prometheus server configuration - When to use NHCB vs standard native histograms - Comparison table ### 3. Working Example ✅ - **New module:** `examples/example-custom-buckets/` - **Demonstrates:** Three real-world use cases: - API latency with arbitrary custom boundaries (0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0 seconds) - Queue size with linear boundaries (10, 20, 30, ..., 100) - Response size with exponential boundaries (100, 1k, 10k, 100k, 1M, 10M bytes) - **Includes:** - Docker Compose setup with Prometheus (NHCB enabled) and Grafana - Pre-configured Grafana dashboard showing all three histogram types - Comprehensive README with build/run instructions and verification steps ### 4. Verification Report ✅ - **File:** `CUSTOM_BUCKETS_VERIFICATION.md` - **Documents:** Detailed findings, test methodology, and verification results ## Test Results [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS All tests pass successfully, confirming that custom bucket support works correctly in all scenarios. ## Key Findings ✅ **Custom bucket support is fully functional** - The Java client correctly handles histograms with custom bucket boundaries - Dual-mode operation (classic + native) works seamlessly with custom buckets - Custom buckets are properly serialized in both text and protobuf formats - Prometheus servers can convert these to NHCB (schema -53) upon ingestion ## How to Test ### Run the tests: ```bash ./mvnw test -Dtest=CustomBucketsHistogramTest -pl prometheus-metrics-core ``` Run the example: ```bash ./mvnw package cd examples/example-custom-buckets docker-compose up ``` Then visit: - Application: http://localhost:9400/metrics - Prometheus: http://localhost:9090 - Grafana: http://localhost:3000 (admin/admin) Documentation Links - https://prometheus.io/docs/specs/native_histograms/ - CUSTOM_BUCKETS_VERIFICATION.md - examples/example-custom-buckets/README.md --------- Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent bcc0639 commit c50c800

File tree

12 files changed

+1335
-5
lines changed

12 files changed

+1335
-5
lines changed

‎CLAUDE.md‎

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -69,19 +69,46 @@ Pre-built instrumentations: `prometheus-metrics-instrumentation-jvm`, `-caffeine
6969
## Code Style
7070

7171
- **Formatter**: Google Java Format (enforced via Spotless)
72-
- **Line length**: 100 characters
72+
- **Line length**: 100 characters (enforced for ALL files including Markdown, Java, YAML, etc.)
7373
- **Indentation**: 2 spaces
7474
- **Static analysis**: Error Prone with NullAway (`io.prometheus.metrics` package)
7575
- **Logger naming**: Logger fields must be named `logger` (not `log`, `LOG`, or `LOGGER`)
7676
- **Assertions in tests**: Use static imports from AssertJ (`import static org.assertj.core.api.Assertions.assertThat`)
7777
- **Empty catch blocks**: Use `ignored` as the exception variable name
78+
- **Markdown code blocks**: Always specify language (e.g., ` ```java`, ` ```bash`, ` ```text`)
7879

7980
## Linting and Validation
8081

81-
- **IMPORTANT**: Always run `mise run build` after modifying Java files to ensure all lints, code formatting (Spotless), static analysis (Error Prone), and checkstyle checks pass
82-
- **IMPORTANT**: Always run `mise run lint:super-linter` after modifying non-Java files (YAML, Markdown, shell scripts, JSON, etc.)
83-
- Super-linter is configured to only show ERROR-level messages via `LOG_LEVEL=ERROR` in `.github/super-linter.env`
84-
- Local super-linter version is pinned to match CI (see `.mise/tasks/lint/super-linter.sh`)
82+
**CRITICAL**: These checks MUST be run before creating any commits. CI will fail if these checks fail.
83+
84+
### Java Files
85+
86+
- **ALWAYS** run `mise run build` after modifying Java files to ensure:
87+
- Code formatting (Spotless with Google Java Format)
88+
- Static analysis (Error Prone with NullAway)
89+
- Checkstyle validation
90+
- Build succeeds (tests are skipped; run `mise run test` or `mise run test-all` to execute tests)
91+
92+
### Non-Java Files (Markdown, YAML, JSON, shell scripts, etc.)
93+
94+
- **ALWAYS** run `mise run lint:super-linter` after modifying non-Java files
95+
- Super-linter will **auto-fix** many issues (formatting, trailing whitespace, etc.)
96+
- It only reports ERROR-level issues (configured via `LOG_LEVEL=ERROR` in `.github/super-linter.env`)
97+
- Common issues caught:
98+
- Lines exceeding 100 characters in Markdown files
99+
- Missing language tags in fenced code blocks
100+
- Table formatting issues
101+
- YAML/JSON syntax errors
102+
103+
### Running Linters
104+
105+
```bash
106+
# After modifying Java files (run BEFORE committing)
107+
mise run build
108+
109+
# After modifying non-Java files (run BEFORE committing)
110+
mise run lint:super-linter
111+
```
85112

86113
## Testing
87114

‎docs/content/getting-started/metric-types.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,94 @@ for [Histogram.Builder](/client_java/api/io/prometheus/metrics/core/metrics/Hist
121121
for a complete list of options. Some options can be configured at runtime,
122122
see [config]({{< relref "../config/config.md" >}}).
123123

124+
### Custom Bucket Boundaries
125+
126+
The default bucket boundaries are designed for measuring request durations in seconds. For other
127+
use cases, you may want to define custom bucket boundaries. The histogram builder provides three
128+
methods for this:
129+
130+
**1. Arbitrary Custom Boundaries**
131+
132+
Use `classicUpperBounds(...)` to specify arbitrary bucket boundaries:
133+
134+
```java
135+
Histogram responseSize = Histogram.builder()
136+
.name("http_response_size_bytes")
137+
.help("HTTP response size in bytes")
138+
.classicUpperBounds(100, 1000, 10000, 100000, 1000000) // bytes
139+
.register();
140+
```
141+
142+
**2. Linear Boundaries**
143+
144+
Use `classicLinearUpperBounds(start, width, count)` for equal-width buckets:
145+
146+
```java
147+
Histogram queueSize = Histogram.builder()
148+
.name("queue_size")
149+
.help("Number of items in queue")
150+
.classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100
151+
.register();
152+
```
153+
154+
**3. Exponential Boundaries**
155+
156+
Use `classicExponentialUpperBounds(start, factor, count)` for exponential growth:
157+
158+
```java
159+
Histogram dataSize = Histogram.builder()
160+
.name("data_size_bytes")
161+
.help("Data size in bytes")
162+
.classicExponentialUpperBounds(100, 10, 5) // 100, 1k, 10k, 100k, 1M
163+
.register();
164+
```
165+
166+
### Native Histograms with Custom Buckets (NHCB)
167+
168+
Prometheus supports a special mode called Native Histograms with Custom Buckets (NHCB) that uses
169+
schema -53. In this mode, custom bucket boundaries from classic histograms are preserved when
170+
converting to native histograms.
171+
172+
The Java client library automatically supports NHCB:
173+
174+
1. By default, histograms maintain both classic (with custom buckets) and native representations
175+
2. The classic representation with custom buckets is exposed to Prometheus
176+
3. Prometheus servers can convert these to NHCB upon ingestion when configured with the
177+
`convert_classic_histograms_to_nhcb` scrape option
178+
179+
Example:
180+
181+
```java
182+
// This histogram will work seamlessly with NHCB
183+
Histogram apiLatency = Histogram.builder()
184+
.name("api_request_duration_seconds")
185+
.help("API request duration")
186+
.classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0) // custom boundaries
187+
.register();
188+
```
189+
190+
On the Prometheus side, configure the scrape job:
191+
192+
```yaml
193+
scrape_configs:
194+
- job_name: "my-app"
195+
scrape_protocols: ["PrometheusProto"]
196+
convert_classic_histograms_to_nhcb: true
197+
static_configs:
198+
- targets: ["localhost:9400"]
199+
```
200+
201+
{{< hint type=note >}}
202+
NHCB is useful when:
203+
204+
- You need precise bucket boundaries for your specific use case
205+
- You're migrating from classic histograms and want to preserve bucket boundaries
206+
- Exponential bucketing from standard native histograms isn't a good fit for your distribution
207+
{{< /hint >}}
208+
209+
See [examples/example-custom-buckets](https://github.com/prometheus/client_java/tree/main/examples/example-custom-buckets) <!-- editorconfig-checker-disable-line -->
210+
for a complete example with Prometheus and Grafana.
211+
124212
Histograms and summaries are both used for observing distributions. Therefore, the both implement
125213
the `DistributionDataPoint` interface. Using the `DistributionDataPoint` interface directly gives
126214
you the option to switch between histograms and summaries later with minimal code changes.

‎examples/example-custom-buckets/README.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
# Native Histograms with Custom Buckets (NHCB) Example
2+
3+
This example demonstrates how to use native histograms with custom bucket boundaries (NHCB) in
4+
Prometheus Java client. It shows three different types of custom bucket configurations and how
5+
Prometheus converts them to native histograms with schema -53.
6+
7+
## What are Native Histograms with Custom Buckets?
8+
9+
Native Histograms with Custom Buckets (NHCB) is a Prometheus feature that combines the benefits of:
10+
11+
- **Custom bucket boundaries**: Precisely defined buckets optimized for your specific use case
12+
- **Native histograms**: Efficient storage and querying capabilities of native histograms
13+
14+
When you configure Prometheus with `convert_classic_histograms_to_nhcb: true`, it converts classic
15+
histograms with custom buckets into native histograms using schema -53, preserving the custom
16+
bucket boundaries.
17+
18+
## Example Metrics
19+
20+
This example application generates three different histogram metrics demonstrating different
21+
bucket configuration strategies:
22+
23+
### 1. API Latency - Arbitrary Custom Boundaries
24+
25+
```java
26+
Histogram apiLatency = Histogram.builder()
27+
.name("api_request_duration_seconds")
28+
.classicUpperBounds(0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0)
29+
.register();
30+
```
31+
32+
**Use case**: Optimized for typical API response times in seconds.
33+
34+
### 2. Queue Size - Linear Boundaries
35+
36+
```java
37+
Histogram queueSize = Histogram.builder()
38+
.name("message_queue_size")
39+
.classicLinearUpperBounds(10, 10, 10) // 10, 20, 30, ..., 100
40+
.register();
41+
```
42+
43+
**Use case**: Equal-width buckets for monitoring queue depth or other discrete values.
44+
45+
### 3. Response Size - Exponential Boundaries
46+
47+
```java
48+
Histogram responseSize = Histogram.builder()
49+
.name("http_response_size_bytes")
50+
.classicExponentialUpperBounds(100, 10, 6) // 100, 1k, 10k, 100k, 1M, 10M
51+
.register();
52+
```
53+
54+
**Use case**: Data spanning multiple orders of magnitude (bytes, milliseconds, etc).
55+
56+
## Build
57+
58+
This example is built as part of the `client_java` project:
59+
60+
```shell
61+
./mvnw package
62+
```
63+
64+
This creates `./examples/example-custom-buckets/target/example-custom-buckets.jar`.
65+
66+
## Run
67+
68+
With the JAR file present, run:
69+
70+
```shell
71+
cd ./examples/example-custom-buckets/
72+
docker-compose up
73+
```
74+
75+
This starts three Docker containers:
76+
77+
- **[http://localhost:9400/metrics](http://localhost:9400/metrics)** - Example application
78+
- **[http://localhost:9090](http://localhost:9090)** - Prometheus server (with NHCB enabled)
79+
- **[http://localhost:3000](http://localhost:3000)** - Grafana (user: _admin_, password: _admin_)
80+
81+
You might need to replace `localhost` with `host.docker.internal` on macOS or Windows.
82+
83+
## Verify NHCB Conversion
84+
85+
### 1. Check Prometheus Configuration
86+
87+
The Prometheus configuration enables NHCB conversion:
88+
89+
```yaml
90+
scrape_configs:
91+
- job_name: "custom-buckets-demo"
92+
scrape_protocols: ["PrometheusProto"]
93+
convert_classic_histograms_to_nhcb: true
94+
scrape_classic_histograms: true
95+
```
96+
97+
### 2. Verify in Prometheus
98+
99+
Visit [http://localhost:9090](http://localhost:9090) and run queries:
100+
101+
```promql
102+
# View histogram metadata (should show schema -53 for NHCB)
103+
prometheus_tsdb_head_series
104+
105+
# Calculate quantiles from custom buckets
106+
histogram_quantile(0.95, rate(api_request_duration_seconds[1m]))
107+
108+
# View raw histogram structure
109+
api_request_duration_seconds
110+
```
111+
112+
### 3. View in Grafana
113+
114+
The Grafana dashboard at [http://localhost:3000](http://localhost:3000) shows:
115+
116+
- p95 and p50 latencies for API endpoints (arbitrary custom buckets)
117+
- Queue size distribution (linear buckets)
118+
- Response size distribution (exponential buckets)
119+
120+
## Key Observations
121+
122+
1. **Custom Buckets Preserved**: The custom bucket boundaries you define are preserved when
123+
converted to NHCB (schema -53).
124+
125+
2. **Dual Representation**: By default, histograms maintain both classic and native
126+
representations, allowing gradual migration.
127+
128+
3. **Efficient Storage**: Native histograms provide more efficient storage than classic histograms
129+
while preserving your custom bucket boundaries.
130+
131+
4. **Flexible Bucket Strategies**: You can choose arbitrary, linear, or exponential buckets based
132+
on your specific monitoring needs.
133+
134+
## When to Use Custom Buckets
135+
136+
Consider using custom buckets (and NHCB) when:
137+
138+
- **Precise boundaries needed**: You know the expected distribution and want specific bucket edges
139+
- **Migrating from classic histograms**: You want to preserve existing bucket boundaries
140+
- **Specific use cases**: Default exponential bucketing doesn't fit your distribution well
141+
- Temperature ranges (might include negative values)
142+
- Queue depths (discrete values with linear growth)
143+
- File sizes (exponential growth but with specific thresholds)
144+
- API latencies (specific SLA boundaries)
145+
146+
## Differences from Standard Native Histograms
147+
148+
| Feature | Standard Native Histograms | NHCB (Schema -53) |
149+
| ----------------- | ------------------------------- | --------------------------------- |
150+
| Bucket boundaries | Exponential (base 2^(2^-scale)) | Custom boundaries |
151+
| Use case | General-purpose | Specific distributions |
152+
| Mergeability | Can merge with same schema | Cannot merge different boundaries |
153+
| Configuration | Schema level (0-8) | Explicit boundary list |
154+
155+
## Cleanup
156+
157+
Stop the containers:
158+
159+
```shell
160+
docker-compose down
161+
```
162+
163+
## Further Reading
164+
165+
<!-- editorconfig-checker-disable -->
166+
<!-- markdownlint-disable MD013 -->
167+
168+
- [Prometheus Native Histograms Specification](https://prometheus.io/docs/specs/native_histograms/)
169+
- [Prometheus Java Client Documentation](https://prometheus.github.io/client_java/)
170+
- [OpenTelemetry Exponential Histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram)

‎examples/example-custom-buckets/docker-compose.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
version: "3"
2+
services:
3+
example-application:
4+
image: eclipse-temurin:25.0.1_8-jre@sha256:9d1d3068b16f2c4127be238ca06439012ff14a8fdf38f8f62472160f9058464a
5+
network_mode: host
6+
volumes:
7+
- ./target/example-custom-buckets.jar:/example-custom-buckets.jar
8+
command:
9+
- /opt/java/openjdk/bin/java
10+
- -jar
11+
- /example-custom-buckets.jar
12+
prometheus:
13+
image: prom/prometheus:v3.9.1@sha256:1f0f50f06acaceb0f5670d2c8a658a599affe7b0d8e78b898c1035653849a702
14+
network_mode: host
15+
volumes:
16+
- ./docker-compose/prometheus.yml:/prometheus.yml
17+
command:
18+
- --enable-feature=native-histograms
19+
- --config.file=/prometheus.yml
20+
grafana:
21+
image: grafana/grafana:12.3.2@sha256:ba93c9d192e58b23e064c7f501d453426ccf4a85065bf25b705ab1e98602bfb1
22+
network_mode: host
23+
volumes:
24+
- ./docker-compose/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/grafana-datasources.yaml
25+
- ./docker-compose/grafana-dashboards.yaml:/etc/grafana/provisioning/dashboards/grafana-dashboards.yaml
26+
- ./docker-compose/grafana-dashboard-custom-buckets.json:/etc/grafana/grafana-dashboard-custom-buckets.json

0 commit comments

Comments
 (0)