High CPU overhead while using spring-cloud-gateway with agent

[123liuziming]

I am evaluating the performance impact of otel java agent on spring-cloud-gateway application, and I found the performance overhead introduced by agent in this scenario to be very high.


CPU usage doubled in my performance test scenario(I do the performance test with&without otel java agent, and send requests to a spring-cloud-gateway application with 1000 QPS).

I 've analyzed the flame graph
flamegraph.zip

The main reason for this performance overhead is because agent use resetOnEachOperator to propagate context on all the operators. But in this scenario, most of the context propagations are useless for users because SCG do not produce any span in those operators. Do we have some ways to avoid those context propagation?

[laurit]

The main reason for this performance overhead is because agent use resetOnEachOperator to propagate context on all the operators.

Just by looking at the flame graph I'm not able to tell that, did you use some other means to reach that conclusion? Looking at the flame graph you are either using the 1.x version of the agent or an old version. Current agent will add less frames to stack traces in reactive code, it may make reading the flame graph easier.

Do we have some ways to avoid those context propagation?

I don't think there is a magic option for that.

If this is a topic you feel passionate about and wish to work on then https://spring.io/blog/2023/03/29/context-propagation-with-project-reactor-2-the-bumpy-road-of-spring-cloud and other posts in the same series may offer ideas.

[123liuziming]

Thanks for your clarification