Add observed_timestamp to list of special fields in normalize_for_stream processor

[flash1293]

We overlooked observed_timestamp as part of #125699 - along with trace_id, span_id and so on, it should stay in the root of the document, not be moved to attributes

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[eyalkoren]

along with trace_id, span_id and so on, it should stay in the root of the document, not be moved to attributes

These fields do not "stay" at the root (unless the document is identified as OTel) - if they exist in a non-OTel document, they would move to attributes. If their ECS counterpart is detected - then they will get its value and be kept at the root.

What are we talking about in this case - is observed_timestamp the semconv field name? What is the ECS counterpart - event.created? event.ingested?

[flash1293]

Right, my bad, I misinterpreted what I saw during a local test - I forgot about list of the criteria for a doc to be considered otel... Closing, sorry for the noise

[eyalkoren]

Nevertheless, wouldn't we want to map either event.created or event.ingested to observed_timestamp?

[flash1293]

Sounds like it makes sense, but the established logs-*.otel-* mappings do not set that as an alias.... @felixbarny what do you think? Should we? Did we consider this before>

[felixbarny]

Right, event.created and ObservedTimestamp do seem equivalent. I'm not sure this was considered before. If we add an alias, we should also add an OTel relation to the event.created field in ECS.

cc @AlexanderWert

[flash1293]

OK, to summarize there would be three places that need to be adjusted:

• Streams conversion logic
• OTel component templates
• ECS relation for otel

Who can make the final decision whether we should do it? I assume it's @AlexanderWert 🙂