Support Dynamically Changing Log Level During Runtime

[jasonmfehr]

Problem Statement
Our app supports changing the logging level dynamically at runtime (without requiring a restart). We want to extend this functionality to also dynamically set the SDK's log level without needing to re-create any Providers. However, this functionality is not supported based on the comments in global_log_handler.h.

Describe the solution you'd like
When our app calls GlobalLogHandler::SetLogLevel, we would like the SDK to change it's logging level without needing to recreate and Providers.

Describe alternatives you've considered
We considered recreating providers when the log level changes but do not want to have the overhead of doing that in a large, multi-threaded application. We also tried changing the log level by calling GlobalLogHandler::SetLogLevel but found that changing from Debug to Info works (the debug messages no longer appear), but changing from Info to Debug only partially works (debug messages for "OTLP TRACE HTTP Exporter" appear but debug messages for "OTLP HTTP Client" do not appear).

Additional context
Add any other context about the feature request here.

[marcalff]

Re:

We want to extend this functionality to also dynamically set the SDK's log level without needing to re-create any Providers. However, this functionality is not supported based on the comments in global_log_handler.h.

The comments in global_log_handler.h alludes to something different.

A signal provider (say, a tracer provider) implementation uses internal logging, so internal logging should be initialized before a signal provider, to avoid crashing in global_log_handler. This is achieved by initializing the global_log_handler singleton before the signal provider singletons.

Once global_log_handler code is initialized, there is no need to destroy and rebuild a signal provider, it will change nothing.

Re:

We also tried changing the log level by calling GlobalLogHandler::SetLogLevel but found that changing from Debug to Info works (the debug messages no longer appear), but changing from Info to Debug only partially works (debug messages for "OTLP TRACE HTTP Exporter" appear but debug messages for "OTLP HTTP Client" do not appear).

Concurrent access to the content of the global log handler singleton, aka, reading the current log level, is racy.

There is only a global variable that holds the log level, but different threads might see a different value in their CPU cache, when it is changed by SetLogLevel().

To make sure the proper log level is used, the code needs to enforce proper concurrent access, either by using a mutex, or by using an atomic load (on GetLogLevel) or store (on SetLogLevel). This is technically possible, the concern (to evaluate) is the runtime overhead that this may introduce.

[jasonmfehr]

I understand the performance concerns whenever mutexes are used. In our case, an eventually consistent approach would be fine. The GlobalLogHandler::SetLogLevel does not need to block until all threads have picked up the new log level. It's fine if each thread takes a few seconds to pick up the new log level.

[lalitb]

As I recall, dynamic level setting should work with acceptable race condition leading to few stale reads. But the underlying HTTP transport layer (CURL) uses its own static logging configuration that doesn't integrate with the global dynamic system.