What should the agent debug log contain?

[trask]

The current debug log is very verbose, and contains lots of things that look ominous but are expected (examples below).

Users who are experiencing a problem often turn on debug logging, and then think these ominous messages are "the problem".

This has happened in our distro enough that I recently updated our distro to only enable exporter debug logging (and a few other things) when debug logging is enabled (with "trace" level logging everything).

@anuraaga mentioned in today's SIG call that this issue has started affecting them also.

I would support -Dotel.javaagent.debug=true only enabling hand-picked (opt-in) list of debug logging, and having some other setting (-Dotel.javaagent.trace=true feels possibly confusing) for getting the flood of debug logging.

But I'd like others thoughts on this.

---

Examples of ominous looking debug logs:

Anything with a stack trace, e.g.

java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
	at io.grpc.netty.shaded.io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
	at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:238)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
        ...

java.lang.ExceptionInInitializerError
	at io.grpc.netty.shaded.io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
        ...

And even some without a stack trace, in particular I've had a couple of users ask about this thinking it meant that the agent was compiled against Java 11 and didn't support Java 8:

Unable to load instrumentation class: io/opentelemetry/javaagent/instrumentation/jetty/v11_0/Jetty11InstrumentationModule
has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime
only recognizes class file versions up to 52.0

[mateuszrzeszutek]

Also linking #2908

It could be useful (on the trace level, or with a dedicated context propagation debugger flag) to log a stack trace every time a new trace is started - it could help people debug context propagation issues.

[iNikem]

First, I think that we have to review io.opentelemetry.javaagent.tooling.LoggingConfigurer and silence some more of those gRPC stack traces.

Second, I totally agree that current debug log is overwhelming and we need some level of logging which is between silence and current debug.

We can either move some things to TRACE level or to move important things to INFO level and make WARN a default logging level.

[trask]

Closing, can revisit later if anyone has renewed interest.