Spans for long running tasks - can we send span events out as they happen?

[adriangb]

Spans fall short when modeling long-running events because they can't be sent until the event finishes. Child spans do get sent, but they lack any information (attributes, start time, etc.) from their parent. Nonetheless, this is still useful because at least you get some information. Unfortunately, this is not true for span events: if you add an event to a span it won't get sent until the span completes. This means that span events are in some ways inferior to child spans or just emitting an out-of-band log with correlation.

It would be really interesting if we could greedily send out span events as they occur. Further, if we could have some sort of semantic convention around an event for "a span was started but is still ongoing" then we could also improve the situation where you can get no information on a long-running span until it completes.

[jack-berg]

We currently have an experimental event API which sends events as log records. There are several open questions related to the event API, but one of them is how they interact with span events? My opinion is that after we work out some of the other issues with the event API and get it stable, that we should deprecate the span event API and encourage folks to directly use the event API. Users would then be able to configure the SDK on whether events recorded in the context of a span are attached to the span as span events when exported, or exported independently as log events, with trace context fields tying back to the related span.

We're working on spinning up an event working group to discuss these questions and more.

[adriangb]

Nice, thanks for the reply!

It sounds like what you are saying is that:

1. Exporting these sort of "span started" events via logs/events makes more sense than as spans.
2. In the future there may be a specific event for this (assuming something like name=span-started,domain=opentelemetry).
3. The SDKs may be configured so that the existing methods on spans use this new event (which would be emitted immediately) instead of appending it to the span and waiting until that span is emitted.

Is this interpretation correct?

[adriangb]

Quick question for you @jack-berg: is there anything in the spec stopping collectors, exporters, etc. from doing something like dropping an event but not the corresponding span or the span but not the corresponding event? If not that seems like it would be an immediate problem in the face of something like sampling.

[jack-berg]

No there is not. There have been discussions about introducing a log sampler concept to parallel trace samplers. If we did such a thing, it would be natural to:

• Introduce a built in sampler which only records log records if the corresponding span is sampled
• Retain all log records by default (for backwards compatibility and to align with user expectations)

Even with the presence of such a sampler, OpenTelemetry gives you ample ability to shoot yourself in the foot. It will always be possible to produce log records where the associated span isn't sampled, or spans where the associated log records are not sampled. This is fine - things should work sensibly by default but artificially restricting this type assumes to much about use cases and would ultimately frustrate users.

[mladjan-gadzic]

is there any movement on this?

[gregbrowndev]

One idea would be to allow the span start "event" to be emitted independently of the span end "event" (I put event in quotes as I know they're not the same as the span events we can attach to a span discussed above, but potentially could be...).

Currently, in the Python OTEL SDK, we use a context manager to wrap a section of code and the span is exported once the context manager exits. However, the two main issues for long-running spans as mentioned are:

• Child spans tracing some section of code within a long-running span will complete and be exported before the parent span (ideal for distributed tracing, maybe less good for tracing local code in the same process). These child spans are not attached to the parent span, as it hasn't been exported yet. Until the parent span is emitted, they appear in the tracing backend (e.g. Tempo/Grafana) as they're own parent spans, polluting the distributed trace (though this could be a Grafana specific issue which could be solved by adding a temporary placeholder).
• Span events aren't visible until the span completes, likewise log-correlation, profile-correlation, metrics and exemplars, etc. all have the same eventual consistency problem. We can't use any of these tracing features until the complete span is exported to the backend.

If a span start record could be exported immediately, and followed up eventually by the end record, it would allow for streaming based use cases and observing traces progressing in real-time (solving both the problems mentioned above).

I'm sure that's a pretty huge change to the spec though, but hope one day it might be possible to achieve.