Why does this subscriber drop events that don't have spans?

[Kinrany]

The most simple setup with tracing_subscriber::fmt prints events that are logged without a span. This subscriber doesn't, I don't quite understand why. Are the tracing models different between tracing and OpenTelemetry somehow? Am I just doing it wrong if I'm getting this result?

[hawkw]

My understanding is that OpenTelemetry does not have a concept of log events that are not part of a span. Perhaps @jtescher can confirm this?

[davidbarsky]

They somewhat do in the form of https://opentelemetry.io/docs/specs/otel/logs/, but to the best my limited knowledge, they are disjoint from SpanEvents.

[jtescher]

Yes you are both correct. The otel trace signal does not have events outside of spans, nor do distributed tracing systems like jaeger and zipkin support these types of events.

The otel logs signal does have this concept but it is not implemented in this crate. There is some work in progress to experiment with adding this support in https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-appender-tracing.

[Victor-N-Suadicani]

How am I supposed to use these crates together, if I want to export both my logs and spans+events ("traces")? I've tried the following, which kinda works but it seems like it duplicates every single event inside spans, since it is exported by both the logger and the tracer. This gives me duplicate logs in my logging solution (Seq in this case).

let client = reqwest::Client::new();

let filter_layer = EnvFilter::from_default_env();

let tracer_provider = TracerProvider::builder()
    .with_batch_exporter(
        SpanExporter::builder()
            .with_http()
            .with_http_client(client.clone())
            .build()
            .unwrap(),
        runtime::Tokio,
    )
    .build();

let logger_provider = LoggerProvider::builder()
    .with_batch_exporter(
        LogExporter::builder()
            .with_http()
            .with_http_client(client)
            .build()
            .unwrap(),
        runtime::Tokio,
    )
    .build();

tracing_subscriber::registry()
    .with(filter_layer)
    .with(tracing_subscriber::fmt::layer())
    .with(OpenTelemetryLayer::new(
        tracer_provider.tracer(env!("CARGO_PKG_NAME")),
    ))
    .with(OpenTelemetryTracingBridge::new(&logger_provider))
    .init();

// Somewhere else in the code...

tracing::info!("This is outside any span, so only exported by the log exporter once (good)");

#[tracing::instrument]
fn example_span() {
    tracing::info!("This is inside a span, so exported TWICE, once by the log exporter and once by the span exporter (bad)");
}

Am I doing something wrong here? If so, how is this supposed to be set up? Or is there some way to avoid this? Or is the duplication expected/desired? Or perhaps is this just not mature enough to handle this yet (which is fine, no blaming intended 😅)?

[VCasecnikovs]

BUMP Discussion

[mladedav]

As the other comments state, when you use trace signals, you need to have an active span to get anything into opentelemetry, otherwise you can use opentelemetey-appender-tracing which works on log records instead of traces. If you use both, you will get both logs and traces which leads to duplication.

If you want to know more, you'll have to ask.

[Victor-N-Suadicani]

If you use both, you will get both logs and traces which leads to duplication.

But this seems undesirable - I feel like it would make more sense to not include the events included via traces in the log records. So you only either get the events as logs or traces, but not both.

And to be fair, I did ask above but with no reply. Is this duplication intended/expected behavior or how do you avoid it? It seems undesirable to me.

[VCasecnikovs]

@mladedav
Thank you for the fast response!

Shouldn't the lost of spans be unintended behaviour?

• A lot of libraries that are using tracing as a log not always wrap all events in a span, this would lead into losing those events if additional library is not used

Is there an option to add a flag that would create spans based on those events?

[mladedav]

Yeah, sorry about that non-reply.

When you're using both opentelemetry logs and traces, then those are two different subscribers. So I don't think there is a good way for them to somehow negotiate which layer handles what. We're collecting both logs and traces where I work even though there is overlap. But honestly it does serve a bit different purposes.

You might try to create some filters that try to check if the filter logs are top level spans and use that.

As far as opentelemetry traces are concerned, I don't think this behaviour is losing any data. And I don't think we should somehow create dummy spans on the library level.