[DOCS-13461] Add APM Processing Pipelines documentation#34787
[DOCS-13461] Add APM Processing Pipelines documentation#34787mervebolatdd merged 5 commits intomasterfrom
Conversation
New page documenting Processing Pipelines, which let users transform, normalize, and enrich span attributes after ingestion without modifying application code. Covers pipeline creation, management, and the two processor types (Remapper and Parser). Also adds a Processing Pipelines entry to the Trace Pipeline index page and the navigation menu.
github-actions
bot
added
Architecture
Preview links (active after the
|
janine-c
left a comment
janine-c
left a comment
There was a problem hiding this comment.
Looks awesome, Brett! I left some comments about some nits, and you can totally decide to deal with them in other PRs or not at all 🙂
| To create a pipeline: | ||
|
|
||
| 1. Navigate to [**APM > Settings > Pipelines**][1]. | ||
| 2. Click **Add Pipeline**. |
There was a problem hiding this comment.
This button shows up greyed out for me. Might be worth mentioning somewhere that this requires a specific permission?
| From the [Pipelines][1] page, you can: | ||
|
|
||
| - Enable or disable individual pipelines | ||
| - Reorder pipelines by dragging them |
There was a problem hiding this comment.
I'm not sure where to bring this up, but it looks like this feature isn't 100% locked down properly for a user without write permissions. If I try to drag a pipeline, it gives me a confirmation dialogue box, and then if I click "move," it only fails then (without a reason, but my guess is the missing permission). I imagine product would want to grey out the drag handles instead 🙂
| - Edit pipelines in draft mode | ||
| - Restrict access with [RBAC][3] | ||
|
|
||
| Disabling a pipeline stops it from processing newly ingested spans. Previously stored spans are not retroactively modified. |
There was a problem hiding this comment.
| Disabling a pipeline stops it from processing newly ingested spans. Previously stored spans are not retroactively modified. | |
| Disabling a pipeline stops it from processing newly ingested spans. It does not retroactively modify previously stored spans. |
Or "previously processed spans"?
|
|
||
| <div class="alert alert-info">Processors can only be applied to <a href="/tracing/trace_explorer/span_tags_attributes/">span attributes, not span tags</a>.</div> | ||
|
|
||
| ### Remapper processor |
There was a problem hiding this comment.
I'm a little confused about how to get from creating pipelines to adding processors. Do you select a pipeline and then add them...?
|
|
||
| To configure Grok parsing rules: | ||
|
|
||
| 1. **Define the parsing attribute and samples**: Select the attribute that you want to parse, and add sample data for the selected attribute. |
There was a problem hiding this comment.
I can't tell what the radio buttons beside the samples in the screenshot do?
| 4. Retention filters and metrics from spans are computed. | ||
| 5. Spans are stored and indexed. | ||
|
|
||
| Reordering pipelines or processors changes processing behavior. |
There was a problem hiding this comment.
This feels like a bit of a non-sequitur because I can't tell how it relates to the preceding information, or what the change is. Does it change the processing behavior as soon as you reorder them? Does it wait until the beginning of the next life cycle to apply the change?
|
|
||
| ## Preprocessed attributes | ||
|
|
||
| Some span attributes are automatically preprocessed by Datadog before pipelines run. For example, [Quantization of APM Data][5] normalizes resource names by default and cannot be disabled. You can define additional pipelines if you need further customization of these attributes. |
There was a problem hiding this comment.
| Some span attributes are automatically preprocessed by Datadog before pipelines run. For example, [Quantization of APM Data][5] normalizes resource names by default and cannot be disabled. You can define additional pipelines if you need further customization of these attributes. | |
| Datadog preprocesses some span attributes before pipelines run. For example, [Quantization of APM Data][5] normalizes resource names by default and cannot be disabled. You can define additional pipelines if you need further customization of these attributes. |
| 1. Spans are ingested. | ||
| 2. Datadog enrichments are applied (infrastructure, CI, source code metadata). | ||
| 3. Processing Pipelines run. | ||
| 4. Retention filters and metrics from spans are computed. | ||
| 5. Spans are stored and indexed. |
There was a problem hiding this comment.
Is there any way to get rid of some of the passive voice here? It's not a huge deal, but I think it does aid conceptual understanding if a user can get some more information on what is performing each process.
- Clarify that attribute changes "apply to" downstream processors - Add brief instruction on how to add a processor to a pipeline - Disambiguate "those" in Remapper constraints sentence - Use active voice for disable behavior and preprocessed attributes
Co-authored-by: Janine Chan <64388808+janine-c@users.noreply.github.com>
Removed note about reordering pipelines affecting processing behavior.
|
Note for Janine and Brett: I'll address the open questions/feedback shared above separately. We need to update the in-app links to start release process and I wanted to unblock EU eng team. |
3 participants
What does this PR do? What is the motivation?
Addresses DOCS-13461
Adds a new documentation page for APM Processing Pipelines, which let users transform, normalize, and enrich span attributes after ingestion without modifying application code. Also adds a Processing Pipelines entry to the Trace Pipeline index page and the navigation menu.
Changes
content/en/tracing/trace_pipeline/processing_pipelines.mdcontent/en/tracing/trace_pipeline/_index.mdwith a Processing Pipelines sectionconfig/_default/menus/main.en.yamlstatic/images/tracing/processing_pipelines/Merge instructions
Merge readiness:
Additional notes
Source material adapted from internal documentation. Content reviewed for Datadog style compliance (Vale clean).