You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*[Delta to Rate processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor#delta-to-rate-processor)
## Notes on Group by Trace and Tail Sampling processors
35
35
36
-
In order to achieve the desired results when using the tailsampling processor and groupbytrace, **do not use a batch processor before these components in a pipeline**. Using a batch processor before these components might separate spans belonging to a same trace. It is important to pay attention to this detail because these components will try to group all the spans belonging to a trace. In the case of the tailsampling processor this will allow for a sampling decision to affect all spans of a trace, creating a full picture of the trace in case it is sampled. A batch processor immediately after these components does not cause any problems and is recommended to properly pre-process data for subsequent exporters.
36
+
In order to achieve the desired results when using the Tail Sampling and Group by Trace processors, **do not use a Batch processor before these components in a pipeline**. Using a Batch processor before these components might separate spans belonging to a same trace. It is important to pay attention to this detail because these components will try to group all the spans belonging to a trace. In the case of the Tail Sampling processor this will allow for a sampling decision to affect all spans of a trace, creating a full picture of the trace in case it is sampled. A Batch processor immediately after these components does not cause any problems and is recommended to properly pre-process data for subsequent exporters.
37
37
38
-
Besides that, you have to tune the `wait_duration` parameter of the groupbytrace processor and `decision_wait` parameter of the tailsampling processor to be greater than or equal to the maximum expected latency of a trace in your system. Again, this will guarantee that spans of a same trace are processed in the same batch.
38
+
Also, you need to make sure that all the spans for a trace are processed in the same collector instances. This is specially important for a collector running in gateway mode.
39
39
40
-
Finally to really limit the number of traces that should be kept in memory, we recommend that you use the groupbytrace processor before the tailsampling processor. The reason why is because the groupbytrace processor implements a limit for the number of traces to be kept in memory while this is not fully implemented in the tailsampling processor.
40
+
Besides that, you have to tune the `wait_duration` parameter of the Group by Trace processor and `decision_wait` parameter of the Tail Sampling processor to be greater than or equal to the maximum expected latency of a trace in your system. Again, this will guarantee that spans of a same trace are processed in the same batch.
41
41
42
-
The groupbytrace processor will drop the oldest trace in case the `num_traces`limit is exceeded. `wait_duration` and `num_traces` should be scaled to consider the expected traffic in the monitored applications. For example, if the maximum expected latency for a request in your application is 10s and the maximum traffic in number of requests per second that your application can have is 1000 requests per second, `wait_duration`should be set to 10s and `num_traces` should be set to at least 10000 (10 * 1000 requests per second). It is highly recommended that you monitor the `otelcol_processor_groupbytrace_traces_evicted` metric from the collector [self telemetry](https://opentelemetry.io/docs/collector/configuration/#service). If the value in the metric is greater than zero, that means that the collector is receiving more traffic than it can handle and you should increase the `num_traces` accordingly.
42
+
Finally to really limit the number of traces that should be kept in memory, we recommend that you use the Group by Trace processor before the Tail Sampling processor. The reason why is because the Group by Trace processor implements a limit for the number of traces to be kept in memory while this is not fully implemented in the Tail Sampling processor.
43
43
44
+
The Group by Trace processor will drop the oldest trace in case the `num_traces` limit is exceeded. `wait_duration` and `num_traces` should be scaled to consider the expected traffic in the monitored applications.
44
45
45
-
Example:
46
-
```
46
+
### Examples
47
+
48
+
If the maximum expected latency for a request in your application is 10s and the maximum traffic in number of requests per second that your application can have is 1000 requests per second, `wait_duration` should be set to 10s and `num_traces` should be set to at least 10000 (10 * 1000 requests per second). It is highly recommended that you monitor the `otelcol_processor_groupbytrace_traces_evicted` metric from the collector [self telemetry](https://opentelemetry.io/docs/collector/configuration/#service). If the value in the metric is greater than zero, that means that the collector is receiving more traffic than it can handle and you should increase the `num_traces` accordingly.
49
+
50
+
51
+
Example from the description above:
52
+
```yaml
47
53
processors:
48
54
groupbytrace:
49
55
wait_duration: 10s
50
-
num_traces: 20000 # Double the max expected traffic (10 * 1000 requests per second)
56
+
num_traces: 20000# Double the max expected traffic (2 * 10 * 1000 requests per second)
51
57
tail_sampling:
52
58
decision_wait: 1s# This value should be smaller than wait_duration
53
59
policies:
@@ -65,3 +71,26 @@ service:
65
71
66
72
```
67
73
74
+
The Tail Sampling processor is very powerful. It is possible to create complex sampling policies by combining policies. For example, to sample traces from a specific path in case of errors you could use the following configuration:
75
+
76
+
```yaml
77
+
processors:
78
+
tail_sampling:
79
+
decision_wait: 1s
80
+
policies:
81
+
- name: and-policy,
82
+
type: and,
83
+
and:
84
+
and_sub_policy:
85
+
- name: test-policy-9,
86
+
type: string_attribute,
87
+
string_attribute:
88
+
key: http.url,
89
+
values: ["\/users"]
90
+
enabled_regex_matching: true
91
+
- name: error-policy,
92
+
type: status_code,
93
+
status_code: {status_codes: [ERROR, UNSET]}
94
+
```
95
+
96
+
It is highly recommended that you take a look into the [README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor/README.md) of this component to understand all the available policies.
0 commit comments