Feature: Add `request_id` in logs #55

kapoorabhishek24 · 2026-01-02T19:25:46Z

GCP logs were missing request/correlation IDs, which made it hard to correlate logs with traces. PR implements the below:

Summary

Add a context-aware OTLP log layer that injects request_id into log records
Centralize request context propagation in tracing spans with a thread-local fallback and request-boundary cleanup.
Normalize request/correlation ID extraction and fallback behavior to keep IDs consistent across logs and traces.

Changes

Replace the OpenTelemetry tracing appender with a custom log layer + formatter.
Store request context in span extensions and thread-local fallback; add Rocket fairing cleanup at request boundaries.
Update request guards to set both IDs and emit them as trace attributes for cross-system correlation.
Simplify endpoint wiring by relying on the Tracing guard for context setup.

Co-authored-by: Copilot <[email protected]>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03aeb04a09

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-01-05T23:55:59Z

rust/lit-core/lit-observability/src/logging/context_layer.rs

+thread_local! {
+    /// Fallback storage for disconnected span scenarios.
+    /// Thread-bound, not task-bound; clear at request boundaries to prevent leakage.
+    static THREAD_REQUEST_CONTEXT: RefCell<Option<RequestContext>> = const { RefCell::new(None) };


Avoid thread-local fallback in async request handling

Using a thread-local fallback for request context means concurrent async requests that happen to run on the same worker thread will overwrite each other’s IDs. In a tokio-based Rocket server, multiple requests can interleave on one thread, so logs from request A can be tagged with request B’s request_id/correlation_id once B calls set_request_context. This is a regression from the prior span-only approach and will misattribute logs under load. Consider using task-local storage or only span extensions for async servers so context stays per-request.

Useful? React with 👍 / 👎.

Nice, switched to a task-local storage keyed by tokio::task::Id instead of thread-local. Span extensions remain the primary lookup, task-local is just a fallback for logs that lose span context.

GTC6244

Looks pretty good, Abhi! Couple of comments that I may be wrong about - hoping this is the case! ;-)

GTC6244 · 2026-01-05T23:57:46Z

rust/lit-core/lit-api-core/src/context/mod.rs

-            extract_correlation_id(req).unwrap_or_else(|| format!("LD-{}", Uuid::new_v4()));
+        let (request_id, correlation_id) = extract_request_and_correlation_ids(req);
+
+        // Generate fallback ID only if BOTH headers are missing


Is this an anticipated situation? I ask, because each node will create a different one, and we'll only be able to trace what's happening on a single node this way.

yeah i don't think we want a fallback ID. leave it empty if there's no ID parsed.

Removed this entirely per Chris's feedback. If no header is provided, we leave it empty.

GTC6244 · 2026-01-06T00:05:44Z

rust/lit-core/lit-api-core/src/context/mod.rs

+
 use crate::error::{EC, Error, Result, conversion_err_code, validation_err_code};

 pub const HEADER_KEY_X_CORRELATION_ID: &str = "X-Correlation-Id";


as far as I can tell, X-Correlation-Id is just used for backwards compatibility, and the JS SDK stuffs in the request-id value.

yes i believe the JS SDK uses X-Request-Id. code is here: https://github.com/LIT-Protocol/js-sdk/blob/6832ff6255f1320946cfba270eeb7e62fd341ba1/packages/lit-client/src/lib/LitNodeClient/LitNodeApi/src/helper/sendNodeRequest.ts#L42

we can standardize on X-Request-Id since that's what the client uses, or we can support both

nevermind - i see you handle both header options below.

Yes, we support both. X-Request-Id is preferred, but we fall back to X-Correlation-Id for older clients. I remember discussing this before.

GTC6244 · 2026-01-06T00:13:06Z

rust/lit-node/lit-node/src/utils/rocket/fairings.rs

+use rocket::fairing::{Fairing, Info, Kind};
+use rocket::{Data, Request, Response};
+
+/// Clears thread-local request context at request start and after response.


Can we do this? I thought ( may be incorrectly ) that Rocket uses tokio async , and that thread-local storage didn't hold up under load. Rocket has a manager for this in Rocket.Request.Request cache, IIRC.

Thanks for pointing this out. Although I switched to task-local keyed by tokio::task::Id. Looked into Rocket's request cache but the tracing layer doesn't have access to the Request object? so went with task-local approach. Also, added a fairing to clear it at request boundaries. what do you think of this?

I was thinking maybe the other way - use rocket's cache from inside the fairing to sync the tracing context ... but you might have the same ( though inverse issue ). At the end of the day ( pardon the expression) if this works in a test environment, I'd give it a 👍 !

glitch003

this looks good - only change i would make is not to make up a fallback request id. if there's no ID parsed, just leave the field empty

glitch003 · 2026-01-06T02:58:34Z

rust/lit-core/lit-api-core/src/context/mod.rs

+
 use crate::error::{EC, Error, Result, conversion_err_code, validation_err_code};

 pub const HEADER_KEY_X_CORRELATION_ID: &str = "X-Correlation-Id";


yes i believe the JS SDK uses X-Request-Id. code is here: https://github.com/LIT-Protocol/js-sdk/blob/6832ff6255f1320946cfba270eeb7e62fd341ba1/packages/lit-client/src/lib/LitNodeClient/LitNodeApi/src/helper/sendNodeRequest.ts#L42

we can standardize on X-Request-Id since that's what the client uses, or we can support both

glitch003 · 2026-01-06T02:59:05Z

rust/lit-core/lit-api-core/src/context/mod.rs

-            extract_correlation_id(req).unwrap_or_else(|| format!("LD-{}", Uuid::new_v4()));
+        let (request_id, correlation_id) = extract_request_and_correlation_ids(req);
+
+        // Generate fallback ID only if BOTH headers are missing


yeah i don't think we want a fallback ID. leave it empty if there's no ID parsed.

glitch003 · 2026-01-06T02:59:29Z

rust/lit-core/lit-api-core/src/context/mod.rs

+
 use crate::error::{EC, Error, Result, conversion_err_code, validation_err_code};

 pub const HEADER_KEY_X_CORRELATION_ID: &str = "X-Correlation-Id";


nevermind - i see you handle both header options below.

github-actions · 2026-01-08T23:00:20Z

PASS [ 43.056s] (3/3) lit_node::test toxiproxy::perf_tests::load_with_no_latency
PASS [ 43.215s] (2/3) lit_node::test toxiproxy::perf_tests::load_with_50ms_latency_single_link
PASS [ 87.899s] (1/3) lit_node::test toxiproxy::perf_tests::load_with_50ms_latency_all_links

glitch003 and others added 10 commits December 9, 2025 10:07

first pass at adding request id to logs

84ed60e

Update rust/lit-node/lit-node/src/main.rs

683d047

Co-authored-by: Copilot <[email protected]>

change how we set the request id on the span

34e4a57

clippy

848d70e

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

8725876

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

49d364c

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

73300f3

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

9412a91

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

6c9afb8

Refactor spans behaviour

abd8dad

kapoorabhishek24 requested review from GTC6244 and glitch003 January 2, 2026 19:25

kapoorabhishek24 added 2 commits January 5, 2026 15:10

Fix fallback ID

f33d293

Add Rocket fairings

3bd44a5

kapoorabhishek24 changed the title ~~Add request_id/correlation_id in logs~~ Feature: Add request_id/ correlation_id in logs Jan 5, 2026

kapoorabhishek24 added 2 commits January 5, 2026 20:09

Remove redundant tracing

3b7489e

optimise tests

03aeb04

kapoorabhishek24 marked this pull request as ready for review January 5, 2026 23:52

chatgpt-codex-connector bot reviewed Jan 5, 2026

View reviewed changes

GTC6244 reviewed Jan 6, 2026

View reviewed changes

glitch003 requested changes Jan 6, 2026

View reviewed changes

kapoorabhishek24 changed the title ~~Feature: Add request_id/ correlation_id in logs~~ Feature: Add request_id in logs Jan 7, 2026

kapoorabhishek24 added 2 commits January 8, 2026 23:32

Address feedback

5de01eb

Remove fallback UUID

a2f2718

kapoorabhishek24 requested review from GTC6244 and glitch003 January 8, 2026 22:47

Merge branch 'master' into feature/NODE-4860-correlation-id-in-logs

abeb19b


		use crate::error::{EC, Error, Result, conversion_err_code, validation_err_code};

		pub const HEADER_KEY_X_CORRELATION_ID: &str = "X-Correlation-Id";

Feature: Add request_id in logs #55

Are you sure you want to change the base?

Feature: Add request_id in logs #55

Conversation

kapoorabhishek24 commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GTC6244 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glitch003 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Feature: Add `request_id` in logs #55

Feature: Add `request_id` in logs #55

kapoorabhishek24 commented Jan 2, 2026 •

edited

Loading