Issue in OTel agent with custom trace header propagator (TextMapPropagator)

[muthuIntern]

Hi All,

I am trying to include custom trace header propagation for OTel agent to use custom headers.
Below is my implementation,

1. I have implement the TextMapPropagator. pls refer to attached CustomHeaderPropagator.txt file
   CustomHeaderPropagator.txt

2. In the src/main/resources/META-INF/services/ directory, create the io.opentelemetry.context.propagation.TextMapPropagator file, which tells the OpenTelemetry agent about your custom propagator. Added following line in the above file com.example.demo.CustomHeaderPropagator

3. Created a JAR with above files. Pls refer to below screenshot
   

4. Running the Java Application with OpenTelemetry Agent
   java -javaagent:opentelemetry-javaagent.jar
   -Dotel.propagators=com.example.demo.CustomHeaderPropagator
   -Dotel.traces.exporter=otlp
   -Dotel.exporter.otlp.endpoint=http://localhost:4317
   -Dotel.exporter.otlp.protocol=grpc
   -Dotel.resource.attributes=service.name=my-java-app
   -cp my-java-app.jar:custom-propagator-1.0-SNAPSHOT.jar
   com.example.MyApplication

5. Getting below error,
   OpenTelemetry Javaagent failed to start
   io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unrecognized value for otel.propagators: com.example.demo.CustomHeaderPropagator. Make sure the artifact including the propagator is on the classpath.
   at io.opentelemetry.sdk.autoconfigure.PropagatorConfiguration.getPropagator(PropagatorConfiguration.java:70)
   at io.opentelemetry.sdk.autoconfigure.PropagatorConfiguration.configurePropagators(PropagatorConfiguration.java:51)
   at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:493)
   at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installOpenTelemetrySdk(OpenTelemetryInstaller.java:29)
   at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:122)
   at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:102)
   at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
   at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
   at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
   at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
   at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
   at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
   at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
   at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)

[laurit]

Agent can be customized with extensions, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension. Try passing the jar that contains your propagator to the agent with -Dotel.javaagent.extensions.

[muthuIntern]

I tried passing my propagator jar as a extention but still getting same error
java -javaagent:opentelemetry-javaagent.jar -Dotel.propagators=com.example.demo.CustomHeaderPropagator -Dotel.javaagent.extensions=custom-propagator-1.0-SNAPSHOT.jar -Dotel.javaagent.debug=true -cp service-a.jar com.example.opentelemetry.otlp.MyApplication

Note: If i don't pass -Dotel.propagators=com.example.demo.CustomHeaderPropagator the default propagator is getting used.

[laurit]

otel.propagators does not accept a propagator class name but a comma separated list of propagator names. TextMapPropagator is not an SPI so having it under META-INF/services will not work. Please see the demo propagator in the sample extension https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/examples/extension/src/main/java/com/example/javaagent/DemoPropagatorProvider.java Besides the TextMapPropagator you'll have to provide a ConfigurablePropagatorProvider implementation and also the META-INF/services file for that.

[muthuIntern]

Thank you very much @laurit . It worked. I was able to generate additional headers and propagate it further.
One last question, Is it possible to use custom headers (key with encoded value) instead of traceparent in OTel agent?

[laurit]

It is, have a look at otel.propagators in https://opentelemetry.io/docs/languages/java/configuration/ Besides w3c traceparent you could use b3 etc. for context propagation. Multiple propagators can be enabled at the same time.

[muthuIntern]

Thanks @laurit .
I've shared below my use case.
We have two different product units and each would like to invoke services from each products for their cross product use case. The challenge here is, one set of products are being adopted by Skywalking framework for tracing and other one is adopted by OTel framework.

Now, when the request come from Product B (OTel) to product A (Skywalking), traceparent header contains the information about trace_id and span_id but skywalking won't understand this header because it is designed to look for sw6 key in headers.
sw6 is a encoded value contains trace_id, span_id and etc.,
I can convert traceparent header value to sw6 and propagate it further. It is working fine as well.

I am exploring custom propagator to solve this conversion every time.
Product B(skwalking) will send sw6 header as it is to Product A (OTel), custom propagator in OTel will have to convert this sw6 to traceparent and wise versa so the we can avoid the conversions from Product A (skywalking).