Adding Tempo for distributed tracing #4556
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Sep 12, 2025
Merged
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ndr-ds
changed the base branch from
09-10-add_more_instrumentation
to
graphite-base/4556
ndr-ds
force-pushed
the
graphite-base/4556
branch
from
e3a8454 to
568a3b0
Compare
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
from
395665d to
1224256
Compare
ndr-ds
changed the base branch from
graphite-base/4556
to
09-10-use_release_max_level_trace_instead
Draft
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
9 times, most recently
from
7131de0 to
d6d672c
Compare
ndr-ds
changed the base branch from
09-10-use_release_max_level_trace_instead
to
graphite-base/4556
This was referenced Sep 15, 2025
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
from
d6d672c to
dd4dd3f
Compare
ndr-ds
force-pushed
the
graphite-base/4556
branch
from
568a3b0 to
3d9a544
Compare
ndr-ds
changed the base branch from
graphite-base/4556
to
09-10-use_release_max_level_trace_instead
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
from
dd4dd3f to
43c36fd
Compare
ndr-ds
force-pushed
the
09-10-use_release_max_level_trace_instead
branch
from
bf70753 to
a7b6f9b
Compare
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
from
ed95829 to
d621078
Compare
ndr-ds
force-pushed
the
09-10-use_release_max_level_trace_instead
branch
from
a7b6f9b to
dfb6602
Compare
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
2 times, most recently
from
17b970e to
39ba3b2
Compare
ndr-ds
force-pushed
the
09-10-use_release_max_level_trace_instead
branch
from
dfb6602 to
dd8730c
Compare
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
2 times, most recently
from
9dcc0e2 to
f62b7d2
Compare
deuszx
reviewed
Sep 18, 2025
deuszx
reviewed
Sep 18, 2025
deuszx
approved these changes
Sep 18, 2025
Contributor
deuszx
left a comment
deuszx
left a comment
There was a problem hiding this comment.
LGTM but let's wait for more input.
ndr-ds
changed the base branch from
09-10-use_release_max_level_trace_instead
to
graphite-base/4556
ndr-ds
force-pushed
the
09-09-adding_tempo_for_open_tracing
branch
from
f62b7d2 to
cfa2f37
Compare
ndr-ds
force-pushed
the
graphite-base/4556
branch
from
dd8730c to
76afd49
Compare
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 22, 2025
## Motivation Now that we have distributed tracing (after #4556), we need more instrumentation so we have data about more functions in the breakdowns. ## Proposal Instrument more functions with `telemetry_only` so that we don't get spammed in our logs, but the spans still get sent to Tempo. ## Test Plan Tested this with #4556, saw the spans properly show in the breakdowns. ## Release Plan - Nothing to do / These changes follow the usual release cycle.
ndr-ds
added a commit
that referenced
this pull request
Sep 22, 2025
Distributed tracing is a great way to debug different types of issues, including for example latency issues. So this is something we definitely want in general, and probably want by default in production as well. Implement Distributed Tracing using Grafana Tempo. As it is a Grafana product, it integrates well with it, which is great for us. The visualizations also seem to be decent. Deployed a network with this code and the `linera-infra` portion of this, and everything works as expected, and I can see the latency breakdowns (I got a really high latency outlier example):  I also chose this because it shows we might be waiting in the chain worker channel's queue for a while here 🤔 which might be worth investigating, which I'll do next. - Nothing to do / These changes follow the usual release cycle.
ndr-ds
added a commit
that referenced
this pull request
Sep 22, 2025
Now that we have distributed tracing (after #4556), we need more instrumentation so we have data about more functions in the breakdowns. Instrument more functions with `telemetry_only` so that we don't get spammed in our logs, but the spans still get sent to Tempo. Tested this with #4556, saw the spans properly show in the breakdowns. - Nothing to do / These changes follow the usual release cycle.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Distributed tracing is a great way to debug different types of issues, including for example latency issues. So this is something we definitely want in general, and probably want by default in production as well.
Proposal
Implement Distributed Tracing using Grafana Tempo. As it is a Grafana product, it integrates well with it, which is great for us. The visualizations also seem to be decent.
Test Plan
Deployed a network with this code and the
linera-infraportion of this, and everything works as expected, and I can see the latency breakdowns (I got a really high latency outlier example):I also chose this because it shows we might be waiting in the chain worker channel's queue for a while here 🤔 which might be worth investigating, which I'll do next.
Release Plan