Skip to content

docs: start doc for distributed tracing and logs guidance #3122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/logs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# OpenTelemetry Rust Logs

Status: **Work-In-Progress**

## Introduction

This document provides guidance on leveraging OpenTelemetry logs
in Rust applications.

## OTel Log Bridge API

The OpenTelemetry Log Bridge API (part of the `opentelemetry` crate) is **not intended
for direct use by application developers**. It is provided for authoring log
appenders that bridge existing logging systems with OpenTelemetry. Bridges for
`tracing` and `log` crates are already available.

## Instrumentation Guidance

1. **Use the `tracing` crate**: We strongly recommend using the
[`tracing`](https://crates.io/crates/tracing) crate for structured logging in
Rust applications.

2. **Explicitly provide `name` and `target` fields**: These map to OpenTelemetry's
EventName and Instrumentation Scope respectively, instead of relying on defaults.

3. **For setup details**: See
[`opentelemetry-appender-tracing`](https://docs.rs/opentelemetry-appender-tracing/)
for mapping details and code examples.

```rust
use tracing::error;
error!(
name: "database_connection_failed",
target: "database",
error_code = "CONNECTION_TIMEOUT",
retry_count = 3,
message = "Failed to connect to database after retries"
);
```

## Terminology

OpenTelemetry defines Events as Logs with an EventName. Since every log from the `tracing`
crate has a `name` field that maps to EventName, every log becomes an OpenTelemetry Event.

**Note**: These are **not** mapped to Span Events. If you want to emit Span Events,
use [`tracing-opentelemetry`](https://docs.rs/tracing-opentelemetry/).

## See Also

- [OpenTelemetry Logs
Specification](https://opentelemetry.io/docs/specs/otel/logs/)
- [`tracing` Documentation](https://docs.rs/tracing/)
- [`opentelemetry-appender-tracing`
Documentation](https://docs.rs/opentelemetry-appender-tracing/)
63 changes: 63 additions & 0 deletions docs/traces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# OpenTelemetry Rust Traces

Status: **Work-In-Progress**

## Introduction

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd have the first sentence have a high-level overview as to what the guidance is—e.g., say something like "use tokio-rs/tracing within a process, use OpenTelemetry at the process boundaries", and then go into detail as below.


This document provides comprehensive guidance on leveraging OpenTelemetry traces
in Rust applications.

## Instrumentation Guidance

1. **Use OTel API for distributed traces**

Use the `opentelemetry::trace` API to create spans. This supports context

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I have this right?

Suggested change
Use the `opentelemetry::trace` API to create spans. This supports context
Use the `opentelemetry::trace` API to create that should be propagated across a process boundary. This supports context

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention here was to recommend OTel API for all spans, not just the ones on edge.
and recommend using tracing:span for contextual log enrichment only.

Of course, this is just the basic scenario, and one most people can just follow if starting from scratch. And this will be just like any other OTel clients like Java or Go etc.

For users who already use tracing:span and want to see them as OTel Spans, then # 4 applies.

In my mind, the ideal state is "For non-edge spans (aka internal spans), users should be able to pick between tracing:span or OTel Tracing API with nearly same effect.

propagation, span kinds (server/client), links, and remote parents.

2. **Use tracing for logs/events**

Use `tracing::info!`, `tracing::event!`, etc. for structured logging. This
will be converted to OTel LogRecords via opentelemetry-appender-tracing and
will be automatically correlated with the current active OTel trace context
as well.

3. **In-proc contextual enrichment for logs/events**

Use `tracing::span!` macros to add contextual metadata (e.g., filename) that
applies to a group of logs. The `otel-appender-tracing` crate will be
enhanced to extract span attributes and attach them to logs automatically.

OpenTelemetry does not have a spec-ed out solution for in-process contextual
enrichment. This is very specific to the logging library (tracing) and its
bridge.

4. **If using tracing::span! to create spans**

This is not directly supported by OpenTelemetry. Use the
`tracing-opentelemetry` bridge to convert tracing spans into OTel spans.

There are some limitations with this approach arising due to `tracing`s lack of support for
creating Spans following OpenTelemetry specification. For example,
`tracing` is unable to:
- Set remote parent
- Specify span kind (e.g., server/client/producer/consumer).
- Add span links

The bridge offers extension APIs to support some of these, but they are not
standard and are maintained outside the OpenTelemetry and Tracing project and
within the bridge itself.

TODO: Should we make a recommendation about
avoiding this extension APIs for instrumentation?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually find the OpenTelemetrySpanExt (I assume you are referring to it) quite useful, and functional in its intended goal of bridging the gap. You can use it effectively for context propagation, setting span kind, adding links and and even dynamic attributes to a span (that are not supported by the pure tracing::span! macro).

I understand that is not standard for open telemetry, but then also the event!, info! etc macros are not standard interfaces in opentelemetry (and also suffer from the same limitation that they don't support dynamic attributes). In my humble opinion, to recommend avoiding altogether the OpenTelemetrySpanExt, which I suspect is widely used in the wild (i certainly use it), is a little premature at this point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing your thoughts!

OpenTelemetrySpanExt (I assume you are referring to it) quite useful,

Of course they are useful and mostly works. I don't think we (OTel) can recommend using it - it's not a standard API, not bound by OTel specs. (I totally understand users use them a lot which is why we need to come up with formal guidance)

but then also the event!, info! etc macros are not standard interfaces in opentelemetry

This is by-design. OTel Rust does not intend to make an end-user facing Logging API, and recommends exiting logging libraries. This is the case in every OTel language implementations. (though some languages have started offering end-user facing logging API (eg: C++), OTel Rust has no such plans.)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, so the direction is to keep a dependency with tokio tracing only for logging, and to add the tracing layer (with tracing-opentelemetry) only if we intend to surface tokio tracing spans (maybe created in other crates) in our traces, otherwise just stick with straight otel apis?

Maybe it would be useful to add a link to examples in this context, I didn't realize that now they are actually good quality and provide a great starting point.

If you are creating spans to track in-proc work (what OTel calls "internal" spans),
`tracing:span` API is sufficient with `tracing-opentelemetry` bridge converting the
`tracing` Span to OTel Span, and properly activating/de-activating OTel's context,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

properly activating/de-activating OTel's context

Is this the right level of detail for a user-facing doc?

to ensure correlation.

5. **Use instrumentation libraries when possible**

If you're manually creating `tracing::span!` and converting to OTel span for
"edge" spans, consider using official instrumentation libraries where
available. These handle proper span creation and context propagation using
the OpenTelemetry API directly.
Loading