You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/sources/tempo/metrics-from-traces/span-metrics/span-metrics-metrics-generator.md
+76-6Lines changed: 76 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,10 +18,11 @@ refs:
18
18
19
19
Part of the metrics-generator, the span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.
20
20
21
-
Span metrics generate two metrics:
21
+
Span metrics generate three metrics:
22
22
23
23
- A counter that computes requests
24
24
- A histogram that tracks the distribution of durations of all requests
25
+
- A counter that tracks the total size of spans ingested
25
26
26
27
Span metrics are of particular interest if your system is not monitored with metrics,
27
28
but it has distributed tracing implemented.
@@ -43,6 +44,26 @@ Refer to [the configuration details](https://grafana.com/docs/tempo/<TEMPO_VERSI
43
44
44
45
If you want to enable metrics-generator for your Grafana Cloud account, refer to the [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/) documentation.
45
46
47
+
### Enabling specific metrics (subprocessors)
48
+
49
+
Instead of enabling all span metrics, you can enable individual metric types using subprocessors in the overrides configuration:
50
+
51
+
-`span-metrics-latency` - Enables only the `traces_spanmetrics_latency` histogram
52
+
-`span-metrics-count` - Enables only the `traces_spanmetrics_calls_total` counter
53
+
-`span-metrics-size` - Enables only the `traces_spanmetrics_size_total` counter
54
+
55
+
Example overrides configuration:
56
+
57
+
```yaml
58
+
overrides:
59
+
defaults:
60
+
metrics_generator:
61
+
processors:
62
+
- span-metrics-latency
63
+
- span-metrics-count
64
+
# span-metrics-size omitted to disable size metrics
65
+
```
66
+
46
67
## How it works
47
68
48
69
The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions.
@@ -52,7 +73,7 @@ This processor mirrored the implementation from the OpenTelemetry Collector of t
52
73
The OTel `spanmetricsprocessor` has since been [deprecated](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/processor/spanmetricsprocessor/README.md) and replaced with the [span metric connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/connector/spanmetricsconnector/README.md).
53
74
54
75
{{< admonition type="note" >}}
55
-
To learn more about cardinality and how to perform a dry run of the metrics generator, see the [Cardinality documentation](ref:cardinality).
76
+
To learn more about cardinality and how to perform a dry run of the metrics generator, refer to the [Cardinality documentation](ref:cardinality).
56
77
{{< /admonition >}}
57
78
58
79
### Metrics
@@ -65,7 +86,9 @@ The following metrics are exported:
65
86
| traces_spanmetrics_calls_total | Counter | Dimensions | Total count of the span |
66
87
| traces_spanmetrics_size_total | Counter | Dimensions | Total size of spans ingested |
67
88
68
-
By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, `status_code`, `status_message`, `job`, and `instance`.
89
+
By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, and `status_code`.
90
+
91
+
The `status_message`, `job`, and `instance` labels are optional and require additional configuration, as described in the sections below.
69
92
70
93
-`service` - The name of the service that generated the span
71
94
-`span_name` - The unique name of the span
@@ -83,19 +106,50 @@ By default, the metrics processor adds the following labels to each metric: `ser
83
106
-`job` - The name of the job, a combination of namespace and service; only added if `metrics_generator.processor.span_metrics.enable_target_info: true`
84
107
-`instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` and `metrics_generator.processor.span_metrics.enable_instance_label: true`
85
108
109
+
### Disabling intrinsic dimensions
110
+
111
+
You can control which intrinsic dimensions are included in your metrics. Disable any of the default intrinsic dimensions using the `intrinsic_dimensions` configuration. This is useful for reducing cardinality when certain labels are not needed.
112
+
113
+
```yaml
114
+
metrics_generator:
115
+
processor:
116
+
span_metrics:
117
+
intrinsic_dimensions:
118
+
service: true
119
+
span_name: true
120
+
span_kind: false # Disable span_kind label
121
+
status_code: true
122
+
status_message: false # Disabled by default
123
+
```
124
+
125
+
### Adding custom dimensions
126
+
86
127
Additional user defined labels can be created using the [`dimensions` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
87
128
When a configured dimension collides with one of the default labels (for example, `status_code`), the label for the respective dimension is prefixed with double underscore (for example, `__status_code`).
88
129
130
+
### Renaming dimensions with dimension_mappings
131
+
89
132
Custom labeling of dimensions is also supported using the [`dimension_mappings` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
133
+
134
+
**Understanding dimensions vs dimension_mappings:**
135
+
136
+
Use `dimensions` when you want to add span attributes as labels using their default (sanitized) names. Use `dimension_mappings` when you want to rename attributes to custom label names or combine multiple attributes.
137
+
138
+
When using `dimension_mappings`, you do not need to also list the same attributes in `dimensions`. The `dimension_mappings` configuration reads directly from the original span attributes.
90
139
You can use `dimension_mappings` to rename a single attribute to a different label name, or to combine multiple attributes into a single composite label.
91
140
92
-
This example shows how to rename the `deployment.environment` attribute to a new label called `deployment_environment`.
141
+
{{< admonition type="note" >}}
142
+
The `source_labels` field must contain the **original span or resource attribute names** (with dots), not sanitized Prometheus label names. For example, use `deployment.environment`, not `deployment_environment`.
143
+
{{< /admonition >}}
144
+
145
+
The `name` field can use either dots (`.`) or underscores (`_`), as both are converted to underscores (`_`) in the final Prometheus metric labels. For example, both `env` and `env.label` result in `env_label` in Prometheus metrics.
146
+
147
+
The following example shows how to rename the `deployment.environment` attribute to a shorter label called `env`, for example:
93
148
94
149
```yaml
95
150
dimension_mappings:
96
-
- name: deployment_environment
151
+
- name: env
97
152
source_labels: ["deployment.environment"]
98
-
join: "-"# Ignored when only one source_label
99
153
```
100
154
101
155
This example shows how to combine the `service.name`, `service.namespace`, and `service.version` attributes into a single label called `service_instance`. The `join` parameter specifies the separator used to join the attribute values together.
@@ -117,6 +171,22 @@ The resulting metric label is `service_instance="abc/def/ghi"`.
117
171
118
172
An optional metric called `traces_target_info` using all resource level attributes as dimensions can be enabled in the [`enable_target_info` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
119
173
174
+
### Excluding dimensions from target_info
175
+
176
+
When `enable_target_info` is enabled, all resource attributes are included as labels on the `traces_target_info` metric. To reduce cardinality, you can exclude specific attributes using the `target_info_excluded_dimensions` configuration:
177
+
178
+
```yaml
179
+
metrics_generator:
180
+
processor:
181
+
span_metrics:
182
+
enable_target_info: true
183
+
target_info_excluded_dimensions:
184
+
- "telemetry.sdk.version"
185
+
- "process.runtime.version"
186
+
```
187
+
188
+
### Handling sampled traces
189
+
120
190
If you use a ratio-based sampler, you can use the custom sampler below to not lose metric information. However, you also need to set `metrics_generator.processor.span_metrics.span_multiplier_key` to `"X-SampleRatio"`.
0 commit comments