Skip to content

docs: start doc for distributed tracing and logs guidance #3122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/logs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# OpenTelemetry Rust Logs

Status: **Work-In-Progress**

## Introduction

This document provides comprehensive guidance on leveraging OpenTelemetry
logs in Rust applications.

## Instrumentation Guidance

// TODO
// Draft points to cover
1. OTel Log-Bridge API is not meant for end-users
2. End users must use existing logging API and bridge them.
3. Recommend `tracing`
4. Recommendation about Name, Target, Message
5. add more.
56 changes: 56 additions & 0 deletions docs/traces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# OpenTelemetry Rust Traces

Status: **Work-In-Progress**

## Introduction

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd have the first sentence have a high-level overview as to what the guidance is—e.g., say something like "use tokio-rs/tracing within a process, use OpenTelemetry at the process boundaries", and then go into detail as below.


This document provides comprehensive guidance on leveraging OpenTelemetry traces
in Rust applications.

## Instrumentation Guidance

1. **Use OTel API for distributed traces**

Use the `opentelemetry::trace` API to create spans. This supports context

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I have this right?

Suggested change
Use the `opentelemetry::trace` API to create spans. This supports context
Use the `opentelemetry::trace` API to create that should be propagated across a process boundary. This supports context

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention here was to recommend OTel API for all spans, not just the ones on edge.
and recommend using tracing:span for contextual log enrichment only.

Of course, this is just the basic scenario, and one most people can just follow if starting from scratch. And this will be just like any other OTel clients like Java or Go etc.

For users who already use tracing:span and want to see them as OTel Spans, then # 4 applies.

In my mind, the ideal state is "For non-edge spans (aka internal spans), users should be able to pick between tracing:span or OTel Tracing API with nearly same effect.

propagation, span kinds (server/client), links, and remote parents.

2. **Use tracing for logs/events**

Use `tracing::info!`, `tracing::event!`, etc. for structured logging. This
will be converted to OTel LogRecords via opentelemetry-appender-tracing and
will be automatically correlated with the current active OTel trace context
as well.

3. **In-proc contextual enrichment for logs/events**

This is not something OTel has a spec-ed out solution for. This is very
specific to the logging library (tracing) and its bridge.

Use `tracing::span!` macros to add contextual metadata (e.g., filename) that
applies to a group of logs. The `otel-appender-tracing` crate will be
enhanced to extract span attributes and attach them to logs automatically.

4. **If using tracing::span! to create spans**

This is not directly supported by OpenTelemetry. Use the
`tracing-opentelemetry` bridge to convert tracing spans into OTel spans.

There are some limitations with this approach arising due to `tracing`s lack of support for
creating Spans following OpenTelemetry specification. Examples include
- Cannot set remote parent
- Cannot specify span kind (e.g., server/client)
- Cannot add span links

The bridge offers extension APIs to support some of these, but they are not
standard and are maintained outside the OpenTelemetry and Tracing project and
within the bridge itself.

TODO: Should we make a recommendation about
avoiding this extension APIs for instrumentation?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually find the OpenTelemetrySpanExt (I assume you are referring to it) quite useful, and functional in its intended goal of bridging the gap. You can use it effectively for context propagation, setting span kind, adding links and and even dynamic attributes to a span (that are not supported by the pure tracing::span! macro).

I understand that is not standard for open telemetry, but then also the event!, info! etc macros are not standard interfaces in opentelemetry (and also suffer from the same limitation that they don't support dynamic attributes). In my humble opinion, to recommend avoiding altogether the OpenTelemetrySpanExt, which I suspect is widely used in the wild (i certainly use it), is a little premature at this point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing your thoughts!

OpenTelemetrySpanExt (I assume you are referring to it) quite useful,

Of course they are useful and mostly works. I don't think we (OTel) can recommend using it - it's not a standard API, not bound by OTel specs. (I totally understand users use them a lot which is why we need to come up with formal guidance)

but then also the event!, info! etc macros are not standard interfaces in opentelemetry

This is by-design. OTel Rust does not intend to make an end-user facing Logging API, and recommends exiting logging libraries. This is the case in every OTel language implementations. (though some languages have started offering end-user facing logging API (eg: C++), OTel Rust has no such plans.)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, so the direction is to keep a dependency with tokio tracing only for logging, and to add the tracing layer (with tracing-opentelemetry) only if we intend to surface tokio tracing spans (maybe created in other crates) in our traces, otherwise just stick with straight otel apis?

Maybe it would be useful to add a link to examples in this context, I didn't realize that now they are actually good quality and provide a great starting point.

5. **Use instrumentation libraries when possible**

If you're manually creating `tracing::span!` and converting to OTel span for
"edge" spans, consider using official instrumentation libraries where
available. These handle proper span creation and context propagation using
the OpenTelemetry API directly.
Loading