Configuring Async Zipkin Exporter for Java Agent

[bmistry13]

Hi All,

I am using the latest Release v1.1.0 java agent to do instrumentation, and I see in the Zipkin call graph that Agent is making many calls to export POST inline (same application thread doing work) for each span. Is it possible to configure Zipkin exporter as ASYN so the application thread does not pay the cost to POST instead it will collect all spans in memory and post Asyncly in batch to the Zipkin server?

here is sample:
http.flavor1.1
http.methodPOST
http.status_code202
http.urlhttp://10.40.43.210:9411/api/v2/spans
net.peer.name10.40.43.210
net.peer.port9411
net.transportIP.TCP
otel.library.nameio.opentelemetry.javaagent.okhttp-3.0
otel.library.version1.1.0
thread.id232
thread.nameOkHttp http://10.40.43.210:9411/...

Please let me know if there any configuration Zipkin exporter that can be done ASYNC.

Thanks,

Bhavesh

[jkwatson]

What SpanProcessor do you have configured? With a BatchSpanProcessor, none of the export work should be being done on the application thread. If you use the SimpleSpanProcessor, it will be done on the same thread that ends that span.

[jkwatson]

Oh, you mention the java agent. The agent should be configuring a BatchSpanProcessor, so I don't think this activity is on the main application thread.

[bmistry13]

How do I conifgure BatchSpanProcessor ? I only see following options in documentation:

java -javaagent:path/to/opentelemetry-javaagent-all.jar
-Dotel.resource.attributes=service.name=your-service-name
-Dotel.traces.exporter=zipkin
-jar myapp.jar

[jkwatson]

the agent will already be using the BatchSpanProcessor. Why do you think that the export is happening on the main thread? I don't think that's possible.

[bmistry13]

@jkwatson ,

I have attached a full trace screen from the Zipkin. You will see POST calls inline. I have NOT able to see this with v 0.14 agnent.

Please let me know if these are ASYNC or Sync calls because this is making the call graph useless because of each span there are POST calls to the Zipkin exporter. I did not see this behavior with an earlier version of the agent as I mentioned in V0.14. This noise makes it hard to read the call graph. So based on the graph, I am interpreting that the agent is making an inline call to POST rather than Async.

Please help to clarify my understanding based on the attached trace call. I appreciate your help!

Thanks,

BHavesh

[jkwatson]

You point at some HTTP posts in that trace, but I can't see how they are related to the zipkin exporter at all. Also, the agent should never be instrumenting the zipkin exporter itself, so I don't think this is what you think it is.

[jkwatson]

Can you post the details of those spans you are alleging are from the zipkin exporter?

[jkwatson]

@bmistry13 Are you enabling the agent on an application that you've also manually configured an SDK with the SimpleSpanProcessor and the Zipkin exporter? You should definitely not have both the agent and your own copy of the SDK running in the same application.

[bmistry13]

Hi @jkwatson ,

Thank you for your quick reply. My use case is very simple: I need both manual and automatic instrumentation of span to be exported, can you please let me know what should one do. You recommended not to use an agent, but I won't get automatic agent instrumented telemetry data if I do not use the agent. How shall I proceed? This must be a common use case where you want manual and automatic together.

Thanks,

Bhavesh

[jkwatson]

There is documentation in the Java instrumentation project about how to do manual instrumentation to augment the agent instrumentation. https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/manual-instrumentation.md

[marcingrzejszczak]

BatchProcessor has a hard dependency on metrics. If you're using tracing without metrics then you'll have class not found exception. I'll need to work this around in Sleuth but something feels wrong about this design (or I'm missing sth).

[bmistry13]

Thank your help and prompt reply. I understand that Java Agent and Spring-Sleuth have been working since the recent Spring-Sleuth update snapshot. The image you see on top was produced by Java Agent and Spring-Sleuth working together. BOTH WORKS only thing is https://github.com/spring-cloud-incubator/spring-cloud-sleuth-otel/blob/832d35a754ca0d11205b95fe83d5077ce58444e5/spring-cloud-sleuth-otel/src/main/java/org/springframework/cloud/sleuth/otel/bridge/Slf4jApplicationListener.java does not receive events hence it is NOT logging in log traceId and spanId. @marcingrzejszczak, is there any way to make it work together with Agent or a way to register an event processor with Agent.

Typically in the dev environment, you run with Agent ON and Sleuth will be on, in production Spring-Sleuth is on. Distributive logs are very important except traceID and spanid is not logged in a log statement. Is there Any way to work together?

I have tried putting it in by myself and it works with Agent but the ONLY issue is span in the log will be static not created by the Agent won't match with Zipkin:

I can added top of FILTER:

boolean traceAdded = false;
try{
                    if(StringUtils.isBlank(traceId)){
                        traceAdded = true;
                        MDC.put(TRACE_ID,ctx.traceId());
                        MDC.put(SPAN_ID,ctx.spanId());
                    }
// do my work()
}finally{
                if(traceAdded){
                    MDC.remove(TRACE_ID);
                    MDC.remove(SPAN_ID);
                }
}

For example,
2021-05-12 15:00:34.972 INFO [core-service,,] 1 --- [ApplianceService-Publish-1] .c.s.i.t.TracePlatformTransactionManager : Creating new span cause a new transaction is started

Another question:

1. How do I check the presence of Java Agent?
2. If Agent is present, How do I register Span Delegator which will fire an event when span start/stop/end so I can fire those events and change MDC value for span id?

Once again thanks for your support so far and prompt reply.

Thanks,

Bhavesh

[jkwatson]

I don't think anyone from the sleuth team nor the otel instrumentation agent team is interested in supporting this mixed-mode operation. This is not a recommended way of working, and we aren't going to guarantee there is interoperability.