Performance and tooling related backports #4626
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Motivation We're adding Pyroscope support back, but using eBPF this time ## Proposal * Add Pyroscope to our helm chart * Add a Grafana dashboard with the profiling info ## Test Plan Deployed a network and saw the Pyroscope profiling data both on the Pyroscope UI and on Grafana, as well as logs from Pyroscope on Loki:    ## Release Plan - Nothing to do / These changes follow the usual release cycle.
## Motivation Optimize RocksDB performance for prefix scans ## Proposal Enhance the RocksDB backend with several performance optimizations: - Add optimized `ReadOptions` for prefix scans with async I/O enabled - Set precise upper bounds for iterators to minimize key traversal - Improve iterator validity checking with a more robust loop structure - Configure bloom filters for prefix iteration optimization - Increase block size from 4KB to 32KB to reduce iterator seeks - Set up prefix extraction for bloom filter optimization - Enable memory-mapped files for faster reads - Configure memtable bloom filters and other performance settings ## Test Plan Tested this with the benchmarks, saw a performance improvement. Not a step change improvement, but significant enough to warrant a PR ## Release Plan Nothing to do / These changes follow the usual release cycle. ## Links - [reviewer checklist](https://github.com/linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
## Motivation If we want to keep the investigation centered on Grafana, it would be useful to also have a memory profiling dashboard containing the flamegraphs ## Proposal Add a dashboard containing the memory profile flamegraphs to Grafana ## Test Plan Will update with a screenshot ## Release Plan - Nothing to do / These changes follow the usual release cycle.
Distributed tracing is a great way to debug different types of issues, including for example latency issues. So this is something we definitely want in general, and probably want by default in production as well. Implement Distributed Tracing using Grafana Tempo. As it is a Grafana product, it integrates well with it, which is great for us. The visualizations also seem to be decent. Deployed a network with this code and the `linera-infra` portion of this, and everything works as expected, and I can see the latency breakdowns (I got a really high latency outlier example):  I also chose this because it shows we might be waiting in the chain worker channel's queue for a while here 🤔 which might be worth investigating, which I'll do next. - Nothing to do / These changes follow the usual release cycle.
Now that we have distributed tracing (after #4556), we need more instrumentation so we have data about more functions in the breakdowns. Instrument more functions with `telemetry_only` so that we don't get spammed in our logs, but the spans still get sent to Tempo. Tested this with #4556, saw the spans properly show in the breakdowns. - Nothing to do / These changes follow the usual release cycle.
deuszx
approved these changes
Sep 23, 2025
Contributor
deuszx
left a comment
deuszx
left a comment
There was a problem hiding this comment.
LGTM.
I wouldn't put the RocksDB optimization in the same backport PR though.
Contributor
|
Let me enable landing without squashing on this branch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This backports a few PRs:
Proposal
Backport the PRs
Test Plan
CI