|
| 1 | +--- |
| 2 | +title: 'Processors' |
| 3 | +description: | |
| 4 | + Processors pre-process the data collected by the receivers before they are exported by exporters. Processors can modify, batch or |
| 5 | + filter the data flowing through the pipeline. |
| 6 | +path: '/docs/components/kafka-receiver-exporter' |
| 7 | +--- |
| 8 | + |
| 9 | +import SectionSeparator from "components/MdxSectionSeparator/sectionSeparator.jsx" |
| 10 | + |
| 11 | +Processors are used in several stages of an OpenTelemetry collector pipeline. They are used to pre-process the data being passed in the pipeline. In a processor the data can be modified, batched, filtered or sampled. The |
| 12 | +ADOT collector supports a selected list of processors. |
| 13 | + |
| 14 | +<SectionSeparator /> |
| 15 | + |
| 16 | +## Processors supported by ADOT collector |
| 17 | + |
| 18 | +The ADOT collector supports the following processors: |
| 19 | + |
| 20 | +* [attributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor#attributes-processor) |
| 21 | +* [batchprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batch-processor) |
| 22 | +* [deltatorateprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor#delta-to-rate-processor) |
| 23 | +* [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor#filter-processor) |
| 24 | +* [groupbytraceprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/groupbytraceprocessor/README.md) |
| 25 | +* [memorylimiterprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor#memory-limiter-processor) |
| 26 | +* [metricsgenerationprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricsgenerationprocessor#metrics-generation-processor) |
| 27 | +* [metricstransformprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor#metrics-transform-processor) |
| 28 | +* [probabilisticsamplerprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#probabilistic-sampling-processor) |
| 29 | +* [resourcedetectionprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#resource-detection-processor) |
| 30 | +* [resourceprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor#resource-processor) |
| 31 | +* [spanprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanprocessor#span-processor) |
| 32 | +* [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor/README.md) |
| 33 | + |
| 34 | +## Notes on Group by Trace and Tail Sampling processors |
| 35 | + |
| 36 | +In order to achieve the desired results when using the tailsampling processor and groupbytrace, **do not use a batch processor before these components in a pipeline**. It is important to pay attention to this detail because these components will try to group all the spans belonging to a trace. In the case of the tailsampling processor this will allow for a sampling decision to affect all spans of a trace, creating a full picture of the trace in case the trace is sampled. A batch processor immediately after these components does not cause any problems and is recommended to properly pre-process data for subsequent exporters. |
| 37 | + |
| 38 | +Besides that, you have to tune the `wait_duration` parameter of the groupbytrace processor and `decision_wait` parameter of the tailsampling processor to be greater than or equal to the maximum expected latency of a trace in your system. Again, this will guarantee that spans of a same trace are processed in the same batch. |
| 39 | + |
| 40 | +Finally to really limit the number of traces that should be kept in memory, we recommend that you use the groupbytrace processor before the tailsampling processor. The reason why is because the groupbytrace processor implements a limit for the number of traces to be kept in memory while this is not fully implemented in the tailsampling processor. |
| 41 | + |
| 42 | +The groupbytrace processor will drop the oldest trace in case the `num_traces` limit is exceeded. `wait_duration` and `num_traces` should be scaled to consider the expected traffic in the monitored applications. For example, if the maximum expected latency for a request in your application is 10s and the maximum traffic in number of requests per second that your application can have is 1000 requests per second, `wait_duration` should be set to 10s and `num_traces` should be set to at least 10000 (10 * 1000 requests per second). It is highly recommended that you monitor the `otelcol_processor_groupbytrace_traces_evicted` metric from the collector [self telemetry](https://opentelemetry.io/docs/collector/configuration/#service). If the value in the metric is greater than zero, that means that the collector is receiving more traffic than it can handle and you should increase the `num_traces` accordingly. |
| 43 | + |
| 44 | + |
| 45 | +Example: |
| 46 | +``` |
| 47 | +processors: |
| 48 | + groupbytrace: |
| 49 | + wait_duration: 10s |
| 50 | + num_traces: 20000 # Double the max expected traffic (10 * 1000 requests per second) |
| 51 | + tail_sampling: |
| 52 | + decision_wait: 1s # This value should be smaller than wait_duration |
| 53 | + policies: |
| 54 | + - ..... # Applicable policies |
| 55 | + batch/traces: |
| 56 | + timeout: 0s # No need to wait more since this will happen in previous processors |
| 57 | + send_batch_max_size: 8196 # This will still allow us to limit the size of the batches sent to subsequent exporters |
| 58 | +
|
| 59 | +service: |
| 60 | + pipelines: |
| 61 | + traces: |
| 62 | + receivers: [otlp] |
| 63 | + processors: [groupbytrace, tail_sampling, batch/traces] |
| 64 | + exporters: [awsxray] |
| 65 | +
|
| 66 | +``` |
| 67 | + |
0 commit comments