How do I add a custom SpanProcessor to an auto instrumented service?

[fuleow]

I'm trying to add certain baggage entries to the server spans in my service that's auto instrumented by the agent. I have baggage propagation working and wrote my own SpanProcessor, but I am having a hard time adding my custom SpanProcessor. I stumbled across #2022 and the test repo is a really good reference of what I'd like to do (it has a DemoSpanProcessor). It's difficult to follow along the discussion in #2022 as there's been quite a bit of refactoring since then.

Can someone please point me in the right direction?

[trask]

hey @fuleow! check out https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/distro

also, we are planning to support this without having to build your own custom distro, follow #2442 for future updates

[pskannusamy]

Hi @fuleow i am having the same issue on implementing the specifics of your case. But i could not really come to a point that works for baggage and span processor. My application is auto-instrumented by agent and I have added Baggage.current()
.toBuilder()
.put("mybag", "mybagvalue")
.build()
.makeCurrent(); in each request filter. But i am not seeing the baggage value on each child spans. Am i missing something or can you please let me know what are the all steps you have done to implement the baggage and span processor.?

[fuleow]

Hey makeCurrent() returns a Scope that's easiest to use in a try-with-resource block. The Javadocs has a good explanation and example. In a filter you would do something like

try(Scope ignored = ctxWithBaggage.makeCurrent()) {
  chain.doFilter(request, response)
}

[pskannusamy]

Hi Fuleow,

Thanks for your response, my question was can i use baggage for one specific attribute to be shown on every span/child span on a trace. And also it is not service to service call only within a single service, the parent span attributes values are not propagated to child span, hence i used baggage.
This is my sequence of actions i have wrote with auto instrumentation and i have enabled
-Dotel.propagators=tracecontext,baggage,b3,b3multi

but still i am not seeing the expected result of cloud.provider attributes to be seen in child spans

[pskannusamy]

Hey makeCurrent() returns a Scope that's easiest to use in a try-with-resource block. The Javadocs has a good explanation and example. In a filter you would do something like

try(Scope ignored = ctxWithBaggage.makeCurrent()) {
  chain.doFilter(request, response)
}

Hi @fuleow how did you confirm that baggage propagator worked for you. i have a similar usecase like yours but i am having an hardtime getting it to validate whether baggage propagation is working. When i set afield/key in baggage on first request and within the service if i try to see if that baggage is available with the key or not using the log but baggage is coming as empty? am i missing something?

[fuleow]

Your SpanProcessor needs to return true for isStartRequired. I used the demo included in this repo to write tests for validation.

[sydneyosbornagility]

Hi there - I had the BaggageSpanProcessor (from the contrib project) working. Then I tried to add an additional custom span processor with its own SPI. I'm using the embed approach, so I have a java-agent jar that has

/extensions:
baggage-extensions.jar
my-custom-extension.jar

I believe my extension is configured correctly. It implements SpanProcessor, and is registered by an implementation of AutoConfigurationCustomizerProvider that looks like

  @Override
  public void customize(final AutoConfigurationCustomizer autoConfiguration) {
    autoConfiguration.addTracerProviderCustomizer(
      (sdkTracerProviderBuilder, configProperties) -> {
        return sdkTracerProviderBuilder.addSpanProcessor(new MySpanProcessor(configProperties));
      }
    );
  }

afaict, my extension is not doing anything, but I have absolutely no idea how to debug that.

One thing I will note is that my custom span processor has a Slf4J logger as I'm trying to get some logging in there to refine it, but I have also provided a slf4j implementation on the java-agent classpath

# Add SLF4J binding to the main agent classpath
RUN unzip -qo slf4j-simple.jar -d javaagent-contents

# Create extensions directory and add the OpenTelemetry extension JARs
RUN mkdir -p javaagent-contents/extensions && \
    cp baggage-extension.jar javaagent-contents/extensions/ && \
    cp my-extension.jar javaagent-contents/extensions/

So, two questions:

1. Should I just have a single SpanProcessor extension that does something like sdkTracerProviderBuilder.addSpanProcessor(new MySpanProcessor(configProperties)).addSpanProcessor(new BaggageSpanProcessor());. Is mine never loading due to having 2 span processor extensions?
2. How can I get some logging around all of this?

[trask]

hi @fuleow, have you tried adding System.out.println in your AutoConfigurationCustomizerProvider constructor? If that's not getting called, probably due to missing or incorrect META-INF/services/...AutoConfigurationCustomizerProvider

[sydneyosbornagility]

Thanks @trask Yes, I have printlns that I do not see. Here is the structure of my custom processor jar. I don't think there's anything wrong with my META-INF stuff.

my-components.jar (181,670 bytes)
├── META-INF/
│   ├── MANIFEST.MF
│   ├── services/
│   │   └── io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider
│   │       └── "com.mycompany.spring.otel.ComponentEnrichmentAutoConfigurationCustomizer"
│   └── spring/
│       └── org.springframework.boot.autoconfigure.AutoConfiguration.imports
├── com/mycompany/spring/otel/
│   ├── ComponentEnrichmentSpanProcessor.class      ← Span Processor
│   ├── ComponentEnrichmentAutoConfigurationCustomizer.class  ← SPI Implementation
│   └── ...other otel classes

[laurit]

If you don't see your printlns then there is a good chance that you have messed up something in the service file. Above you have put the content of the service file in quotes, which it shouldn't be, idk whether you really have quotes there or you just added them there. If possible consider providing a sample project that reproduces your issue. If we can build and run it it will be much easier to determine why it doesn't work as you expect.

but I have also provided a slf4j implementation on the java-agent classpath

In the agent we use java util logging api, you might wish to consider using that too in your extension. We rewrite the usages of java util logging in the agent (and also in the extensions) and redirect the logging to slf4j simple that is bundled in the agent. The slf4j simple that is bundled in the agent is shaded and the package is relocated. We don't currently rewrite slf4j usages (like we do for java util logging) in the extensions so it isn't usable for them. If you wish to use slf4j inside your extension you may need to shade it inside your extension along with slf4j api. You said that you currently only try to add slf4j-simple.jar into the agent, it could very well be the cause of your issues since without adding the slf4j api your classes that depend on slf4j could be unusable.

[sydneyosbornagility]

Thanks, @laurit. I think part of the issue maybe related to how images are being pulled during our deployments. It's quite possible that my changes were not even being pulled )-: I'll report back once I've resolved that and provide a repro project if I'm still having issues.

[sydneyosbornagility]

OK, I have satisfied myself that the latest image with javaagent jar containing my extension is being pulled. I still do not see my logging. I modified my extension to use java.util logging only. What is the best way to provide the sample project?

[laurit]

github repo would be the best. zip file would also be fine.

[sydneyosbornagility]

Here's a zip file. Thank you for looking.
otel-sample.zip

[trask]

hi @sydneyosbornagility, took a quick look though didn't try running it, I'd suggest trying to whittle it down as much as possible while still exhibiting the failure, e.g.

• remove all the dependencies
• make ComponentEnrichmentSpanProcessor super minimal
• use System.out even

[laurit]

@sydneyosbornagility I built your extension and ran it with

ltulmin@LTULMIN-M-F7RP opentelemetry-java-instrumentation % java -javaagent:/path/to/opentelemetry-java-instrumentation/javaagent/build/libs/opentelemetry-javaagent-2.21.0-SNAPSHOT.jar -Dotel.javaagent.extensions=/path/to/otel-extension-sample/target/ar-otel-0.0.2-SNAPSHOT.jar -version
[otel.javaagent 2025-10-10 22:38:53:387 +0300] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.21.0-SNAPSHOT
*** LOADING CUSTOMIZER CLASS ***
[otel.javaagent 2025-10-10 22:38:53:560 +0300] [main] INFO com.ar.otel.ComponentEnrichmentAutoConfigurationCustomizer - Loading ComponentEnrichmentAutoConfigurationCustomizer class
*** REGISTERING CUSTOMIZER ***
[otel.javaagent 2025-10-10 22:38:53:561 +0300] [main] INFO com.ar.otel.ComponentEnrichmentAutoConfigurationCustomizer - Registering ComponentEnrichmentSpanProcessor with AutoConfigurationCustomizer
*** SPAN PROCESSOR CREATED ***
[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - Initializing ComponentEnrichmentSpanProcessor with ConfigProperties
[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - otel.service.name from configProperties: null

[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - otel.service.name from system property: null

[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - OTEL_SERVICE_NAME from environment: null

[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - spring.application.name from system property: null

[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - otel.resource.attributes: null

[otel.javaagent 2025-10-10 22:38:53:744 +0300] [main] INFO com.ar.otel.ComponentEnrichmentSpanProcessor - Configuring ComponentEnrichmentSpanProcessor with service name: null

openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode)

Looks like your extension works.

[sydneyosbornagility]

@laurit thank you, that's great. It does seem to be working for me now as well, so I'm not sure what bit I took out that made the difference, but I'll take it. One last question. Is there any way to make the java agent output structured logging?