Adding Tempo for distributed tracing (linera-io#4556)

Distributed tracing is a great way to debug different types of issues,
including for example latency issues. So this is something we definitely
want in general, and probably want by default in production as well.

Implement Distributed Tracing using Grafana Tempo. As it is a Grafana
product, it integrates well with it, which is great for us. The
visualizations also seem to be decent.

Deployed a network with this code and the `linera-infra` portion of
this, and everything works as expected, and I can see the latency
breakdowns (I got a really high latency outlier example):

![Screenshot 2025-09-16 at
13.48.59.png](https://app.graphite.dev/user-attachments/assets/98f49272-d04a-4b7e-aa83-c04f90ec7347.png)

I also chose this because it shows we might be waiting in the chain
worker channel's queue for a while here 🤔 which might be worth
investigating, which I'll do next.

- Nothing to do / These changes follow the usual release cycle.

Author: ndr-ds

Cargo.lock

@@ -157,6 +157,14 @@ num-format = "0.4.4"
 num-traits = "0.2.18"
 octocrab = "0.42.1"
 oneshot = "0.1.6"
+opentelemetry = { version = "0.30.0", features = ["trace"] }
+opentelemetry-http = "0.30.0"
+opentelemetry-otlp = { version = "0.30.0", features = [
+    "grpc-tonic",
+    "trace",
+    "tls-roots",
+] }
+opentelemetry_sdk = { version = "0.30.0", features = ["trace", "rt-tokio"] }
 papaya = "0.1.5"
 pathdiff = "0.2.1"
 port-selector = "0.1.6"
@@ -250,6 +258,7 @@ tonic-web-wasm-client = "0.6.0"
 tower = "0.4.13"
 tower-http = "0.6.6"
 tracing = { version = "0.1.40", features = ["release_max_level_debug"] }
+tracing-opentelemetry = "0.31.0"
 tracing-subscriber = { version = "0.3.18", default-features = false, features = [
     "env-filter",
 ] }

Cargo.toml

@@ -25,7 +25,7 @@ ARG binaries=
 ARG copy=${binaries:+_copy}
 ARG build_flag=--release
 ARG build_folder=release
-ARG build_features=scylladb,metrics,memory-profiling
+ARG build_features=scylladb,metrics,memory-profiling,tempo
 ARG rustflags="-C force-frame-pointers=yes"
 
 FROM rust:1.74-slim-bookworm AS builder

docker/Dockerfile

@@ -154,3 +154,10 @@ releases:
     set:
       - name: crds.enabled
         value: "true"
+  - name: tempo
+    version: 1.23.3
+    namespace: tempo
+    chart: grafana/tempo
+    timeout: 900
+    values:
+      - {{ env "LINERA_HELMFILE_VALUES_LINERA_CORE" | default "values-local.yaml.gotmpl" }}

kubernetes/linera-validator/helmfile.yaml

@@ -18,6 +18,12 @@ workspace = true
 metrics = ["prometheus"]
 reqwest = ["dep:reqwest"]
 revm = []
+tempo = [
+    "opentelemetry",
+    "opentelemetry-otlp",
+    "opentelemetry_sdk",
+    "tracing-opentelemetry",
+]
 test = ["test-strategy", "proptest"]
 web = [
     "getrandom/js",
@@ -80,6 +86,10 @@ tracing-web = { optional = true, workspace = true }
 
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
 chrono.workspace = true
+opentelemetry = { workspace = true, optional = true }
+opentelemetry-otlp = { workspace = true, optional = true }
+opentelemetry_sdk = { workspace = true, optional = true }
+tracing-opentelemetry = { workspace = true, optional = true }
 rand = { workspace = true, features = ["getrandom", "std", "std_rng"] }
 tokio = { workspace = true, features = [
     "process",