You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may only be represented as '{}' when not part of a multi-block statement (4.1.3)"/>
139
+
value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may
140
+
only be represented as '{}' when not part of a multi-block statement (4.1.3)"/>
140
141
<messagekey="ws.notPreceded"
141
142
value="WhitespaceAround: ''{0}'' is not preceded with whitespace."/>
Copy file name to clipboardExpand all lines: docs/content/getting-started/metric-types.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -112,8 +112,9 @@ most important options:
112
112
-`classicBuckets(...)`: Set the classic bucket boundaries. Default buckets are `.005`, `.01`,
113
113
`.025`, `.05`, `.1`, `.25`, `.5`, `1`, `2.5`, `5`, `and 10`. The default bucket boundaries are
114
114
designed for measuring request durations in seconds.
115
-
-`nativeMaxNumberOfBuckets()`: Upper limit for the number of native histogram buckets. Default is 160. When the maximum is reached, the native histogram automatically reduces resolution to stay
116
-
below the limit.
115
+
-`nativeMaxNumberOfBuckets()`: Upper limit for the number of native histogram buckets.
116
+
Default is 160. When the maximum is reached, the native histogram automatically
117
+
reduces resolution to stay below the limit.
117
118
118
119
See Javadoc
119
120
for [Histogram.Builder](/client_java/api/io/prometheus/metrics/core/metrics/Histogram.Builder.html)
40
+
<!-- editorconfig-checker-disable-line -->
35
41
36
42
If you move the mouse over an Exemplar, an overlay pops up with a link to a tracing tool
37
43
like [Tempo](https://github.com/grafana/tempo).
38
44
39
45
## Automatic Exemplar Generation
40
46
41
-
The Prometheus Java client library automatically detects OpenTelemetry tracing. The library automatically produces
47
+
The Prometheus Java client library automatically detects OpenTelemetry tracing. The library
48
+
automatically produces
42
49
Exemplars with OpenTelemetry trace IDs if applicable.
43
50
44
51
## Sampling
45
52
46
-
Distributed tracing produces a lot of data, and it is quite common to apply a sampling strategy to reduce the number of
53
+
Distributed tracing produces a lot of data, and it is quite common to apply a sampling strategy to
54
+
reduce the number of
47
55
traces.
48
56
49
57
There are two ways how to do this:
@@ -52,15 +60,21 @@ There are two ways how to do this:
52
60
2._Tail-based sampling_ is performed by an external infrastructure component like
53
61
the [OpenTelemetry collector](https://opentelemetry.io/docs/collector/).
54
62
55
-
When generating Exemplars, the Prometheus Java client libray must make sure that the Exemplar points to a trace that is
56
-
actually sampled. Otherwise, Exemplar's trace ID will not be available in the monitoring backend, i.e. the Exemplar will
63
+
When generating Exemplars, the Prometheus Java client libray must make sure that the Exemplar points
64
+
to a trace that is
65
+
actually sampled. Otherwise, Exemplar's trace ID will not be available in the monitoring backend,
66
+
i.e. the Exemplar will
57
67
link to a non-existing trace ID.
58
68
59
-
With _head-based sampling_ this works out-of-the-box: The Prometheus client library calls OpenTelemetry's
60
-
`Span.current().getSpanContext().isSampled()` function and selects Exemplars only if the current Span is sampled.
69
+
With _head-based sampling_ this works out-of-the-box: The Prometheus client library calls
70
+
OpenTelemetry's
71
+
`Span.current().getSpanContext().isSampled()` function and selects Exemplars only if the current
72
+
Span is sampled.
61
73
62
-
With _tail-based sampling_ there is a trick: The Prometheus client library adds an attribute `exemplar="true"` to each
63
-
Span that is used as an Exemplar. The remainder of this readme shows how you can configure the `tail_sampling` processor
74
+
With _tail-based sampling_ there is a trick: The Prometheus client library adds an attribute
75
+
`exemplar="true"` to each
76
+
Span that is used as an Exemplar. The remainder of this readme shows how you can configure the
77
+
`tail_sampling` processor
64
78
of the OpenTelemetry collector to sample Spans marked with `exemplar="true"`.
65
79
66
80
## Build the Example
@@ -71,50 +85,61 @@ The example is built as part of the `client_java` parent project with
71
85
./mvnw package
72
86
```
73
87
74
-
This should generate two Java services `.../example-greeting-service/target/example-greeting-service.jar` and
88
+
This should generate two Java services
89
+
`.../example-greeting-service/target/example-greeting-service.jar` and
The `docker-compose` file will run the example with the following containers:
80
95
81
-
-`hello-world-app`: Java service described above, with the OpenTelemetry Java instrumentation agent attached for
96
+
-`hello-world-app`: Java service described above, with the OpenTelemetry Java instrumentation agent
97
+
attached for
82
98
distrubted tracing.
83
-
-`greeting-service`: Java service described above, with the OpenTelemetry Java instrumentation agent attached for
99
+
-`greeting-service`: Java service described above, with the OpenTelemetry Java instrumentation
100
+
agent attached for
84
101
distrubted tracing.
85
-
-`collector`: OpenTelemetry collector for receiving the distributed traces, performing _tail sampling_, and forwarding
102
+
-`collector`: OpenTelemetry collector for receiving the distributed traces, performing _tail
103
+
sampling_, and forwarding
86
104
them to `tempo`.
87
-
-`prometheus`: Prometheus server scraping metrics from `hello-world-app`, `greeting-service`, and `tempo`.
105
+
-`prometheus`: Prometheus server scraping metrics from `hello-world-app`, `greeting-service`, and
106
+
`tempo`.
88
107
-`tempo`: Trace database.
89
-
-`grafana`: Has `prometheus` and `tempo` configured as data sources, and is configured with an example dashboard with
108
+
-`grafana`: Has `prometheus` and `tempo` configured as data sources, and is configured with an
109
+
example dashboard with
90
110
Exemplars enabled.
91
-
-`k6`: The [k6](https://k6.io/) load test tool for generating 50 requests / second on the Java services.
111
+
-`k6`: The [k6](https://k6.io/) load test tool for generating 50 requests / second on the Java
112
+
services.
92
113
93
-
With the Java services described above available in the `target/` directories, you can run the example with:
114
+
With the Java services described above available in the `target/` directories, you can run the
115
+
example with:
94
116
95
117
```shell
96
118
cd example-exemplars-tail-sampling
97
119
docker-compose up
98
120
```
99
121
100
-
Grafana will run on [http://localhost:3000](ttp://localhost:3000). The default user is _admin_ with password _admin_.
122
+
Grafana will run on [http://localhost:3000](ttp://localhost:3000). The default user is _admin_ with
123
+
password _admin_.
101
124
102
125
## Example Dashboard
103
126
104
127
The example dashboard shows 50 requests / second for the Java serices:
105
128
106
-
129
+
<!-- editorconfig-checker-disable-line -->
107
130
108
131
The Tempo metrics show that only ~5 traces / second are received:
109
132
110
-
133
+
<!-- editorconfig-checker-disable-line -->
111
134
112
-
The reason is that the OpenTelemetry collector is configured to sample only 10% of the traces. Yet, all Exemplars in the
135
+
The reason is that the OpenTelemetry collector is configured to sample only 10% of the traces. Yet,
136
+
all Exemplars in the
113
137
latency graph point to traces that actually made it to Tempo.
114
138
115
139
## OpenTelemetry Collector Config
116
140
117
-
The OpenTelemetry collector is configured to keep all Spans with the attribute `exemplar="true"`. Probabilistic sampling
141
+
The OpenTelemetry collector is configured to keep all Spans with the attribute `exemplar="true"`.
142
+
Probabilistic sampling
118
143
applies only to the remainder of the Spans.
119
144
120
145
```yaml
@@ -133,17 +158,22 @@ processors:
133
158
]
134
159
```
135
160
136
-
The Prometheus Java client library automatically sets the `exemplar="true"` attribute for each Span that is used as an
161
+
The Prometheus Java client library automatically sets the `exemplar="true"` attribute for each Span
162
+
that is used as an
137
163
Exemplar.
138
164
139
-
The Exemplar sampler of the Prometheus Java client library is rate-limited, the `DEFAULT_MIN_RETENTION_PERIOD_SECONDS`
165
+
The Exemplar sampler of the Prometheus Java client library is rate-limited, the
166
+
`DEFAULT_MIN_RETENTION_PERIOD_SECONDS`
140
167
is 7s.
141
168
So only one Span per 7 seconds per time series will be marked with `exemplar="true"`.
142
169
143
170
## Conclusion
144
171
145
-
The Prometheus Java client library automatically generates Exemplars if OpenTelemetry tracing is detected.
146
-
In order to make that work with _tail-based sampling_, the Prometheus Java client library adds an attribute
172
+
The Prometheus Java client library automatically generates Exemplars if OpenTelemetry tracing is
173
+
detected.
174
+
In order to make that work with _tail-based sampling_, the Prometheus Java client library adds an
175
+
attribute
147
176
`exemplar="true"` to each Span that's used as an Exemplar.
148
-
The `tail_sampling` processor in the OpenTelemetry collector can be configured to always sample Spans marked as
177
+
The `tail_sampling` processor in the OpenTelemetry collector can be configured to always sample
0 commit comments