-
Notifications
You must be signed in to change notification settings - Fork 557
docs: start doc for distributed tracing and logs guidance #3122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# OpenTelemetry Rust Logs | ||
|
||
Status: **Work-In-Progress** | ||
|
||
## Introduction | ||
|
||
This document provides comprehensive guidance on leveraging OpenTelemetry | ||
logs in Rust applications. | ||
|
||
## Instrumentation Guidance | ||
|
||
// TODO | ||
// Draft points to cover | ||
1. OTel Log-Bridge API is not meant for end-users | ||
2. End users must use existing logging API and bridge them. | ||
3. Recommend `tracing` | ||
4. Recommendation about Name, Target, Message | ||
5. add more. |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,56 @@ | ||||||
# OpenTelemetry Rust Traces | ||||||
|
||||||
Status: **Work-In-Progress** | ||||||
|
||||||
## Introduction | ||||||
|
||||||
This document provides comprehensive guidance on leveraging OpenTelemetry traces | ||||||
in Rust applications. | ||||||
|
||||||
## Instrumentation Guidance | ||||||
|
||||||
1. **Use OTel API for distributed traces** | ||||||
|
||||||
Use the `opentelemetry::trace` API to create spans. This supports context | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do I have this right?
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The intention here was to recommend OTel API for all spans, not just the ones on edge. Of course, this is just the basic scenario, and one most people can just follow if starting from scratch. And this will be just like any other OTel clients like Java or Go etc. For users who already use In my mind, the ideal state is "For non-edge spans (aka internal spans), users should be able to pick between |
||||||
propagation, span kinds (server/client), links, and remote parents. | ||||||
|
||||||
2. **Use tracing for logs/events** | ||||||
|
||||||
Use `tracing::info!`, `tracing::event!`, etc. for structured logging. This | ||||||
will be converted to OTel LogRecords via opentelemetry-appender-tracing and | ||||||
will be automatically correlated with the current active OTel trace context | ||||||
as well. | ||||||
|
||||||
3. **In-proc contextual enrichment for logs/events** | ||||||
|
||||||
This is not something OTel has a spec-ed out solution for. This is very | ||||||
specific to the logging library (tracing) and its bridge. | ||||||
|
||||||
Use `tracing::span!` macros to add contextual metadata (e.g., filename) that | ||||||
cijothomas marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
applies to a group of logs. The `otel-appender-tracing` crate will be | ||||||
enhanced to extract span attributes and attach them to logs automatically. | ||||||
|
||||||
4. **If using tracing::span! to create spans** | ||||||
|
||||||
This is not directly supported by OpenTelemetry. Use the | ||||||
`tracing-opentelemetry` bridge to convert tracing spans into OTel spans. | ||||||
|
||||||
There are some limitations with this approach arising due to `tracing`s lack of support for | ||||||
creating Spans following OpenTelemetry specification. Examples include | ||||||
cijothomas marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- Cannot set remote parent | ||||||
- Cannot specify span kind (e.g., server/client) | ||||||
- Cannot add span links | ||||||
|
||||||
The bridge offers extension APIs to support some of these, but they are not | ||||||
standard and are maintained outside the OpenTelemetry and Tracing project and | ||||||
within the bridge itself. | ||||||
|
||||||
TODO: Should we make a recommendation about | ||||||
avoiding this extension APIs for instrumentation? | ||||||
|
||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I actually find the OpenTelemetrySpanExt (I assume you are referring to it) quite useful, and functional in its intended goal of bridging the gap. You can use it effectively for context propagation, setting span kind, adding links and and even dynamic attributes to a span (that are not supported by the pure tracing::span! macro). I understand that is not standard for open telemetry, but then also the event!, info! etc macros are not standard interfaces in opentelemetry (and also suffer from the same limitation that they don't support dynamic attributes). In my humble opinion, to recommend avoiding altogether the OpenTelemetrySpanExt, which I suspect is widely used in the wild (i certainly use it), is a little premature at this point. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for sharing your thoughts!
Of course they are useful and mostly works. I don't think we (OTel) can recommend using it - it's not a standard API, not bound by OTel specs. (I totally understand users use them a lot which is why we need to come up with formal guidance)
This is by-design. OTel Rust does not intend to make an end-user facing Logging API, and recommends exiting logging libraries. This is the case in every OTel language implementations. (though some languages have started offering end-user facing logging API (eg: C++), OTel Rust has no such plans.) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. got it, so the direction is to keep a dependency with tokio tracing only for logging, and to add the tracing layer (with tracing-opentelemetry) only if we intend to surface tokio tracing spans (maybe created in other crates) in our traces, otherwise just stick with straight otel apis? Maybe it would be useful to add a link to examples in this context, I didn't realize that now they are actually good quality and provide a great starting point. |
||||||
5. **Use instrumentation libraries when possible** | ||||||
|
||||||
If you're manually creating `tracing::span!` and converting to OTel span for | ||||||
"edge" spans, consider using official instrumentation libraries where | ||||||
available. These handle proper span creation and context propagation using | ||||||
the OpenTelemetry API directly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd have the first sentence have a high-level overview as to what the guidance is—e.g., say something like "use
tokio-rs/tracing
within a process, use OpenTelemetry at the process boundaries", and then go into detail as below.