Skip to content

Commit 895d75f

Browse files
[DOC] Clarify dimension_mappings config for span metrics (#6383)
* Clarify dimension_mappings config for span metrics * Stylistic updates to align with writers' toolkit * Update from Carles' comment in the ticket
1 parent fb4e083 commit 895d75f

File tree

1 file changed

+76
-6
lines changed

1 file changed

+76
-6
lines changed

docs/sources/tempo/metrics-from-traces/span-metrics/span-metrics-metrics-generator.md

Lines changed: 76 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,11 @@ refs:
1818

1919
Part of the metrics-generator, the span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.
2020

21-
Span metrics generate two metrics:
21+
Span metrics generate three metrics:
2222

2323
- A counter that computes requests
2424
- A histogram that tracks the distribution of durations of all requests
25+
- A counter that tracks the total size of spans ingested
2526

2627
Span metrics are of particular interest if your system is not monitored with metrics,
2728
but it has distributed tracing implemented.
@@ -43,6 +44,26 @@ Refer to [the configuration details](https://grafana.com/docs/tempo/<TEMPO_VERSI
4344

4445
If you want to enable metrics-generator for your Grafana Cloud account, refer to the [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/) documentation.
4546

47+
### Enabling specific metrics (subprocessors)
48+
49+
Instead of enabling all span metrics, you can enable individual metric types using subprocessors in the overrides configuration:
50+
51+
- `span-metrics-latency` - Enables only the `traces_spanmetrics_latency` histogram
52+
- `span-metrics-count` - Enables only the `traces_spanmetrics_calls_total` counter
53+
- `span-metrics-size` - Enables only the `traces_spanmetrics_size_total` counter
54+
55+
Example overrides configuration:
56+
57+
```yaml
58+
overrides:
59+
defaults:
60+
metrics_generator:
61+
processors:
62+
- span-metrics-latency
63+
- span-metrics-count
64+
# span-metrics-size omitted to disable size metrics
65+
```
66+
4667
## How it works
4768

4869
The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions.
@@ -52,7 +73,7 @@ This processor mirrored the implementation from the OpenTelemetry Collector of t
5273
The OTel `spanmetricsprocessor` has since been [deprecated](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/processor/spanmetricsprocessor/README.md) and replaced with the [span metric connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/connector/spanmetricsconnector/README.md).
5374

5475
{{< admonition type="note" >}}
55-
To learn more about cardinality and how to perform a dry run of the metrics generator, see the [Cardinality documentation](ref:cardinality).
76+
To learn more about cardinality and how to perform a dry run of the metrics generator, refer to the [Cardinality documentation](ref:cardinality).
5677
{{< /admonition >}}
5778

5879
### Metrics
@@ -65,7 +86,9 @@ The following metrics are exported:
6586
| traces_spanmetrics_calls_total | Counter | Dimensions | Total count of the span |
6687
| traces_spanmetrics_size_total | Counter | Dimensions | Total size of spans ingested |
6788

68-
By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, `status_code`, `status_message`, `job`, and `instance`.
89+
By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, and `status_code`.
90+
91+
The `status_message`, `job`, and `instance` labels are optional and require additional configuration, as described in the sections below.
6992

7093
- `service` - The name of the service that generated the span
7194
- `span_name` - The unique name of the span
@@ -83,19 +106,50 @@ By default, the metrics processor adds the following labels to each metric: `ser
83106
- `job` - The name of the job, a combination of namespace and service; only added if `metrics_generator.processor.span_metrics.enable_target_info: true`
84107
- `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` and `metrics_generator.processor.span_metrics.enable_instance_label: true`
85108

109+
### Disabling intrinsic dimensions
110+
111+
You can control which intrinsic dimensions are included in your metrics. Disable any of the default intrinsic dimensions using the `intrinsic_dimensions` configuration. This is useful for reducing cardinality when certain labels are not needed.
112+
113+
```yaml
114+
metrics_generator:
115+
processor:
116+
span_metrics:
117+
intrinsic_dimensions:
118+
service: true
119+
span_name: true
120+
span_kind: false # Disable span_kind label
121+
status_code: true
122+
status_message: false # Disabled by default
123+
```
124+
125+
### Adding custom dimensions
126+
86127
Additional user defined labels can be created using the [`dimensions` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
87128
When a configured dimension collides with one of the default labels (for example, `status_code`), the label for the respective dimension is prefixed with double underscore (for example, `__status_code`).
88129

130+
### Renaming dimensions with dimension_mappings
131+
89132
Custom labeling of dimensions is also supported using the [`dimension_mappings` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
133+
134+
**Understanding dimensions vs dimension_mappings:**
135+
136+
Use `dimensions` when you want to add span attributes as labels using their default (sanitized) names. Use `dimension_mappings` when you want to rename attributes to custom label names or combine multiple attributes.
137+
138+
When using `dimension_mappings`, you do not need to also list the same attributes in `dimensions`. The `dimension_mappings` configuration reads directly from the original span attributes.
90139
You can use `dimension_mappings` to rename a single attribute to a different label name, or to combine multiple attributes into a single composite label.
91140

92-
This example shows how to rename the `deployment.environment` attribute to a new label called `deployment_environment`.
141+
{{< admonition type="note" >}}
142+
The `source_labels` field must contain the **original span or resource attribute names** (with dots), not sanitized Prometheus label names. For example, use `deployment.environment`, not `deployment_environment`.
143+
{{< /admonition >}}
144+
145+
The `name` field can use either dots (`.`) or underscores (`_`), as both are converted to underscores (`_`) in the final Prometheus metric labels. For example, both `env` and `env.label` result in `env_label` in Prometheus metrics.
146+
147+
The following example shows how to rename the `deployment.environment` attribute to a shorter label called `env`, for example:
93148

94149
```yaml
95150
dimension_mappings:
96-
- name: deployment_environment
151+
- name: env
97152
source_labels: ["deployment.environment"]
98-
join: "-" # Ignored when only one source_label
99153
```
100154

101155
This example shows how to combine the `service.name`, `service.namespace`, and `service.version` attributes into a single label called `service_instance`. The `join` parameter specifies the separator used to join the attribute values together.
@@ -117,6 +171,22 @@ The resulting metric label is `service_instance="abc/def/ghi"`.
117171

118172
An optional metric called `traces_target_info` using all resource level attributes as dimensions can be enabled in the [`enable_target_info` configuration option](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#metrics-generator).
119173

174+
### Excluding dimensions from target_info
175+
176+
When `enable_target_info` is enabled, all resource attributes are included as labels on the `traces_target_info` metric. To reduce cardinality, you can exclude specific attributes using the `target_info_excluded_dimensions` configuration:
177+
178+
```yaml
179+
metrics_generator:
180+
processor:
181+
span_metrics:
182+
enable_target_info: true
183+
target_info_excluded_dimensions:
184+
- "telemetry.sdk.version"
185+
- "process.runtime.version"
186+
```
187+
188+
### Handling sampled traces
189+
120190
If you use a ratio-based sampler, you can use the custom sampler below to not lose metric information. However, you also need to set `metrics_generator.processor.span_metrics.span_multiplier_key` to `"X-SampleRatio"`.
121191

122192
```go

0 commit comments

Comments
 (0)