Exporter Configurability/Deferred Registration

[FrankSpitulski]

Use case: configure an exporter (and ideally other otel properties) with the spring environment, including remote properties present after the bootstrap phase

I have a hacky implementation as follows and can provide snippets if desired:
There is a proxy span exporter module to give to otel during the agent bootstrap whose jar is shaded and contains the real exporter as well. There is another module containing spring bootstrap config and regular config beans which imports both the auto instrumentation jar and the proxy exporter jar. The bootstrap bean grabs the File path to each of the two jars, extracting them to temp files if necessary (ie, inside a jar already). BytebuddyAgent is used to attach the auto instrumentation at this point, also setting some ota system properties based on the spring bootstrap config. The agent loads the proxy exporter and continues on with the spring initialisation. The proxy exporter will return bad status codes for all operations until the real exporter is loaded. Each operation will attempt to load the real exporter until it is finally loaded. The regular config bean will eventually load and send the properties to the proxy exporter using java serialisation to cross the class loader boundary. When the properties are all set, the next proxy exporter invocation will trigger the real exporter to load (since it needs to load in the exporter's classloader). Future invocations will passthrough and invoke the real exporter. As an aside, I've also included a sort of span mapper in the proxy exporter to map over exported spans and modify them to mark 4xx server errors's status as OK.

The effect of the above is that I do not need to set any javaagent arg, provide any external jars, or set any system properties, special config files, or arguments aside from my usual spring config.

My ideal implementation:
I'd like to be able to programatically provide an exporter like the manual instrumentation OpenTelemetrySdk.getTracerProvider().addSpanProcessor(...) where the exporter was created inside the application without needing to pass properties and without having to select from an included set of exporters. Since I am providing the span processor here, there shouldn't be any defaults still running such as the in memory exporter.

[trask]

Wow, nice setup 😄. It sounds like you are already doing #691, which is something we are interested in also.

Calling OpenTelemetrySdk.getTracerProvider().addSpanProcessor(...) directly from application code is difficult for us, because we have to bridge everything between class loaders, and so the span processor will be less efficient.

In it's place, we want to make it easy to build your own opentelemetry-javaagent "custom distro" jar file, which could include your own span processor etc.

And then I think the additional piece we would still be missing is some way to configure your custom distro via standard spring properties. We could probably watch for those properties via bytecode instrumentation and maybe a configuration callback instead of needing to pass them via system properties.

[trask]

Closing, OpenTelemetry Spring Boot starter is our current recommendation for nice integration with spring properties, and if there are instrumentations lacking please open an issue so that we can prioritize those.

There is also work being done on supporting remote configuration (see #12251).