|
| 1 | +--- |
| 2 | +title: 'Processors' |
| 3 | +description: | |
| 4 | + Processors pre-process the data collected by the receivers before they are exported by exporters. Processors can modify, batch or |
| 5 | + filter the data flowing through the pipeline. |
| 6 | +path: '/docs/components/processors' |
| 7 | +--- |
| 8 | + |
| 9 | +import SectionSeparator from "components/MdxSectionSeparator/sectionSeparator.jsx" |
| 10 | + |
| 11 | +Processors are used in several stages of an OpenTelemetry collector pipeline. They are used to pre-process the data being passed in the pipeline. In a processor the data can be modified, batched, filtered or sampled. The |
| 12 | +ADOT collector supports a selected list of processors. |
| 13 | + |
| 14 | +<SectionSeparator /> |
| 15 | + |
| 16 | +## Processors supported by ADOT collector |
| 17 | + |
| 18 | +The ADOT collector supports the following processors: |
| 19 | + |
| 20 | +* [Attributes processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor#attributes-processor) |
| 21 | +* [Batch processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batch-processor) |
| 22 | +* [Delta to Rate processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor#delta-to-rate-processor) |
| 23 | +* [Filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor#filter-processor) |
| 24 | +* [Group by Trace processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/groupbytraceprocessor/README.md) |
| 25 | +* [Memory Limiter processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor#memory-limiter-processor) |
| 26 | +* [Metrics Generation processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricsgenerationprocessor#metrics-generation-processor) |
| 27 | +* [Metrics Transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor#metrics-transform-processor) |
| 28 | +* [Probabilistic Sampling processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#probabilistic-sampling-processor) |
| 29 | +* [Resource Detection processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#resource-detection-processor) |
| 30 | +* [Resource processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor#resource-processor) |
| 31 | +* [Span processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanprocessor#span-processor) |
| 32 | +* [Tail Sampling processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor/README.md) |
| 33 | + |
| 34 | +## Notes on Group by Trace and Tail Sampling processors |
| 35 | + |
| 36 | +In order to achieve the desired results when using the Tail Sampling and Group by Trace processors, **do not use a Batch processor before these components in a pipeline**. Using a Batch processor before these components might separate spans belonging to a same trace. It is important to pay attention to this detail because these components will try to group all the spans belonging to a trace. In the case of the Tail Sampling processor this will allow for a sampling decision to affect all spans of a trace, creating a full picture of the trace in case it is sampled. A Batch processor immediately after these components does not cause any problems and is recommended to properly pre-process data for subsequent exporters. |
| 37 | + |
| 38 | +Also, you need to make sure that all the spans for a trace are processed in the same collector instances. This is specially important for a collector running in gateway mode. |
| 39 | + |
| 40 | +Besides that, you have to tune the `wait_duration` parameter of the Group by Trace processor and `decision_wait` parameter of the Tail Sampling processor to be greater than or equal to the maximum expected latency of a trace in your system. Also, be sure to include a grace period for network latency between an application and collector. Again, this will guarantee that spans of a same trace are processed in the same batch. |
| 41 | + |
| 42 | +Finally to really limit the number of traces that should be kept in memory, we recommend that you use the Group by Trace processor before the Tail Sampling processor. The reason why is because the Group by Trace processor implements a limit for the number of traces to be kept in memory while this is not fully implemented in the Tail Sampling processor. |
| 43 | + |
| 44 | +The Group by Trace processor will drop the oldest trace in case the `num_traces` limit is exceeded. `wait_duration` and `num_traces` should be scaled to consider the expected traffic in the monitored applications. |
| 45 | + |
| 46 | +### Examples |
| 47 | + |
| 48 | +If the maximum expected latency for a request in your application is 10s and the maximum traffic in number of requests per second that your application can have is 1000 requests per second, `wait_duration` should be set to 10s and `num_traces` should be set to at least 10000 (10 * 1000 requests per second). It is highly recommended that you monitor the `otelcol_processor_groupbytrace_traces_evicted` metric from the collector [self telemetry](https://opentelemetry.io/docs/collector/configuration/#service). If the value in the metric is greater than zero, that means that the collector is receiving more traffic than it can handle and you should increase the `num_traces` accordingly. |
| 49 | + |
| 50 | + |
| 51 | +Example from the description above: |
| 52 | +```yaml |
| 53 | +processors: |
| 54 | + groupbytrace: |
| 55 | + wait_duration: 10s |
| 56 | + num_traces: 20000 # Double the max expected traffic (2 * 10 * 1000 requests per second) |
| 57 | + tail_sampling: |
| 58 | + decision_wait: 1s # This value should be smaller than wait_duration |
| 59 | + policies: |
| 60 | + - ..... # Applicable policies |
| 61 | + batch/tracesampling: |
| 62 | + timeout: 0s # No need to wait more since this will happen in previous processors |
| 63 | + send_batch_max_size: 8196 # This will still allow us to limit the size of the batches sent to subsequent exporters |
| 64 | + |
| 65 | +service: |
| 66 | + pipelines: |
| 67 | + traces/tailsampling: |
| 68 | + receivers: [otlp] |
| 69 | + processors: [groupbytrace, tail_sampling, batch/tracesampling] |
| 70 | + exporters: [awsxray] |
| 71 | + |
| 72 | +``` |
| 73 | + |
| 74 | +The Tail Sampling processor has the functionality to combine sampling policies. For example, to sample traces from a specific path in case of errors you could use the following configuration: |
| 75 | + |
| 76 | +```yaml |
| 77 | +processors: |
| 78 | + tail_sampling: |
| 79 | + decision_wait: 1s |
| 80 | + policies: |
| 81 | + - name: and-policy |
| 82 | + type: and |
| 83 | + and: |
| 84 | + and_sub_policy: |
| 85 | + - name: path-policy |
| 86 | + type: string_attribute |
| 87 | + string_attribute: |
| 88 | + key: http.url |
| 89 | + values: ["\/users"] |
| 90 | + enabled_regex_matching: true |
| 91 | + - name: error-policy |
| 92 | + type: status_code |
| 93 | + status_code: |
| 94 | + status_codes: ["ERROR", "UNSET"] |
| 95 | +``` |
| 96 | +
|
| 97 | +In the next example we will sample 20% of the spans that present an error: |
| 98 | +
|
| 99 | +```yaml |
| 100 | +processors: |
| 101 | + tail_sampling: |
| 102 | + decision_wait: 1s |
| 103 | + policies: |
| 104 | + - name: and-policy |
| 105 | + type: and |
| 106 | + and: |
| 107 | + and_sub_policy: |
| 108 | + - name: error-policy |
| 109 | + type: status_code |
| 110 | + status_code: |
| 111 | + status_codes: ["ERROR", "UNSET"] |
| 112 | + - name: probabilistic-policy |
| 113 | + type: probabilistic |
| 114 | + probabilistic: |
| 115 | + sampling_percentage: 20 |
| 116 | +``` |
| 117 | +
|
| 118 | +To see the full set of policy options available to the tail sampling processor please refer to it's [README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor/README.md). |
0 commit comments