Severe performance problems in 1.54.0 and 1.55.0

[ML-Marco]

Tracer Version(s)

1.54.0, 1.55.0, maybe earlier versions

Java Version(s)

25.0.1

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

I have already contacted support about this but thought it might be a good idea to give this some more visibility here so others can add important observations that could help fix this issue.

We updated from 1.51.0 to 1.54.0 and saw a huge performance decrease in our applications. We rolled back and tried again with 1.55.0 but the problem remains. CPU almost doubled in all Java applications. Memory also increased a bit but the main problem is the added CPU load.

Here is the CPU Time chart for one of our applications:

As mentioned above, there is no problem in 1.51.0. We didn't test the releases up to 1.54.0 so these releases might be affected as well.

This is our observation so far:

Tracing agent	-Ddd.profiling.ddprof.cstack	Performance problems?
1.51.0	unset	No
1.51.0	vm	No
1.54.0	unset	Yes
1.55.0	vm	Yes

Expected Behavior

Same or better performance as in 1.51.0

Reproduction Code

No response

[brunolellis]

Same issue here.
Seeing CPU increase after upgrading to Java 25 (Zulu) and Datadog agent to 1.53.0 and/or 1.54.0. I haven't had a chance to test 1.55.0 yet.
Profiling shows QueryObfuscator.obfuscate(String) and com.google.re2j.* consuming most of the extra CPU in both tested versions.

[dougqh]

Thank you for the report, we are looking into the increased CPU usage.

[brunolellis]

as a workaround, setting the env var DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP to an empty String works (source).

[ML-Marco]

as a workaround, setting the env var DD_TRACE_OBFUSCATION_QUERY_STRING_REGEXP to an empty String works (source).

Maybe yours is another issue.

In my case, when setting -Ddd.profiling.ddprof.enabled=false the increased CPU usage is gone as well, which indicates that it's a problem with the profiler.