docs: start doc for distributed tracing and logs guidance

[cijothomas]

WIP. We already have some docs for metrics, this is adding mostly placeholder for logs, and some guidance into traces.
Not intended to be running examples, but more like setting direction on how to instrument. This is usually not required in OTel repos (as OTel is the obvious API), but we need to cover this to help users navigate the existing ecosystem around tracing.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.5%. Comparing base (c7e47de) to head (640aa63).

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3122     +/-   ##
=======================================
- Coverage   80.5%   80.5%   -0.1%     
=======================================
  Files        126     126             
  Lines      22218   22218             
=======================================
- Hits       17890   17889      -1     
- Misses      4328    4329      +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[davidbarsky]

Do I have this right?

Suggested change

	Use the `opentelemetry::trace` API to create spans. This supports context
	Use the `opentelemetry::trace` API to create that should be propagated across a process boundary. This supports context

[cijothomas]

The intention here was to recommend OTel API for all spans, not just the ones on edge.
and recommend using tracing:span for contextual log enrichment only.

Of course, this is just the basic scenario, and one most people can just follow if starting from scratch. And this will be just like any other OTel clients like Java or Go etc.

For users who already use tracing:span and want to see them as OTel Spans, then # 4 applies.

In my mind, the ideal state is "For non-edge spans (aka internal spans), users should be able to pick between tracing:span or OTel Tracing API with nearly same effect.

[davidbarsky]

I'd have the first sentence have a high-level overview as to what the guidance is—e.g., say something like "use tokio-rs/tracing within a process, use OpenTelemetry at the process boundaries", and then go into detail as below.

[alessandrobologna]

I actually find the OpenTelemetrySpanExt (I assume you are referring to it) quite useful, and functional in its intended goal of bridging the gap. You can use it effectively for context propagation, setting span kind, adding links and and even dynamic attributes to a span (that are not supported by the pure tracing::span! macro).

I understand that is not standard for open telemetry, but then also the event!, info! etc macros are not standard interfaces in opentelemetry (and also suffer from the same limitation that they don't support dynamic attributes). In my humble opinion, to recommend avoiding altogether the OpenTelemetrySpanExt, which I suspect is widely used in the wild (i certainly use it), is a little premature at this point.

[cijothomas]

Thanks for sharing your thoughts!

OpenTelemetrySpanExt (I assume you are referring to it) quite useful,

Of course they are useful and mostly works. I don't think we (OTel) can recommend using it - it's not a standard API, not bound by OTel specs. (I totally understand users use them a lot which is why we need to come up with formal guidance)

but then also the event!, info! etc macros are not standard interfaces in opentelemetry

This is by-design. OTel Rust does not intend to make an end-user facing Logging API, and recommends exiting logging libraries. This is the case in every OTel language implementations. (though some languages have started offering end-user facing logging API (eg: C++), OTel Rust has no such plans.)

[alessandrobologna]

got it, so the direction is to keep a dependency with tokio tracing only for logging, and to add the tracing layer (with tracing-opentelemetry) only if we intend to surface tokio tracing spans (maybe created in other crates) in our traces, otherwise just stick with straight otel apis?

Maybe it would be useful to add a link to examples in this context, I didn't realize that now they are actually good quality and provide a great starting point.

[scottgerring]

properly activating/de-activating OTel's context

Is this the right level of detail for a user-facing doc?