Use release_max_level_trace instead

[ndr-ds]

Motivation

We currently compile out trace logs, even though the overhead is minimal (it's 2 CPU instructions https://www.reddit.com/r/rust/comments/x9nypb/comment/inutkiv/)

Proposal

Stop compiling it out, so we can have trace logs if we want for debugging with just a restart, without having to recompile the binary.

Test Plan

I'm benchmarking networks with this, saw no visible performance regression

Release Plan

• Nothing to do / These changes follow the usual release cycle.

[ndr-ds]

• Reconnect on max_age GoAway #4557
• Use release_max_level_trace instead #4554 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Twey]

What does max_level_trace do, concretely? As far as I know there's no class of logs more verbose than trace 🤔

[ma2bd]

Not sure about this. It's also the difference between tracing and debugging

[ndr-ds]

@Twey It actually does nothing 😅 i.e. it's the same as if we remove it, I think.
@ma2bd if we're sure that we'll never need trace level logs in production, then maybe this isn't needed, but that hasn't been the case in some of the investigations we've done. Also, why not just have it available? There's no overhead, and we're still protected by whatever we set RUST_LOG to be 🤔

[deuszx]

Can you compare binary sizes?

[ma2bd]

What I'm saying is that one could very well define "trace" as the logs that you will never use in production. Perhaps because they are too slow, or perhaps for security/privacy reasons.

Then, "debug" would be the logs you usually don't activate in production, but you could do so for debugging. Etc.

[Twey]

What I'm saying is that one could very well define "trace" as the logs that you will never use in production. Perhaps because they are too slow, or perhaps for security/privacy reasons.

It's impossible to tell in advance what logs will definitely never be needed to debug an issue that may arise in production. If we know for sure that a log line will never be needed in production, I'd suggest we just don't write it (or at least never commit it to main).

Then, "debug" would be the logs you usually don't activate in production, but you could do so for debugging. Etc.

Unfortunately that leaves us in a very awkward place with respect to INFO. We want to be able to do at least some level of retrospective debugging, at least on our internal validators. If DEBUG is turned off by default, that means that we have to either overwhelm INFO (the log level of interest to people who don't know/care about Linera codebase internals) with internal debugging details and references to concepts that end users shouldn't have to care about, or accept that we won't be able to figure out what happened to a service in the past without trying to reproduce the issue with DEBUG turned on.

[deuszx]

I am of an opinion that TRACE is for logs that are very spammy and give full coverage of the flows. Rarely turned on on production deployments due to the amount of logs produced.

Fortunately, log levels can be set selectively on the per-module base as well. So we could say: linera=INFO,linera::storage=TRACE.