report NoSuchMethodError when execute instrumentation:test

[huange7]

i add some code in Instrumenter.class:

private Context doStart(Context parentContext, REQUEST request, @Nullable Instant startTime) {
    // This is my code, and it will not cause any errors when executed at this time.
    if (TracerHelper.shouldInterrupt(instrumentationName)) {
      return parentContext;
    }
    SpanKind spanKind = spanKindExtractor.extract(request);

    // This is my code, and executing this method at this time will result in a NoSuchMethodError.
    InstrumenterHelper InstrumenterHelper =
        TracerHelper.decideTheTracerAndGetHelper(
            tracer, parentContext, spanNameExtractor.extract(request));
    SpanBuilder spanBuilder = InstrumenterHelper.getSpanBuilder();
    parentContext = tencentInstrumenterHelper.getParentContext();

    spanBuilder.setSpanKind(spanKind);
    ...
}

The TracerHelper is in another module, at the same directory level as the "instrumentation-api".

I have included my module in test-common:

api(project(":my-instrumentation:my-api"))

There is a strange phenomenon occurring: when I execute ./gradlew instrumentation:test, some tests result in a NoSuchMethodError. Moreover, TracerHelper has two methods, and when executing shouldInterrupt, there is no error, but when executing decideTheTracerAndGetHelper, an error occurs.

here is the stack:

java.lang.NoSuchMethodError: 'InstrumenterHelper TracerHelper.decideTheTracerAndGetHelper(io.opentelemetry.api.trace.Tracer, io.opentelemetry.context.Context, java.lang.String)'
	at io.opentelemetry.instrumentation.api.instrumenter.Instrumenter.doStart(Instrumenter.java:171)
	at io.opentelemetry.instrumentation.api.instrumenter.Instrumenter.start(Instrumenter.java:134)
	at io.opentelemetry.instrumentation.testing.TestInstrumenters.runWithInstrumenter(TestInstrumenters.java:100)
	at io.opentelemetry.instrumentation.testing.TestInstrumenters.runWithHttpClientSpan(TestInstrumenters.java:73)
	at io.opentelemetry.instrumentation.testing.InstrumentationTestRunner.runWithHttpClientSpan(InstrumentationTestRunner.java:232)
	at io.opentelemetry.instrumentation.testing.junit.http.AbstractHttpClientTest.shouldSuppressNestedClientSpanIfAlreadyUnderParentClientSpan(AbstractHttpClientTest.java:181)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[laurit]

When agent is build opentelemetry-api and instrumentation-api classes are renamed to avoid conflicts when application includes the same classes. Probably your own TracerHelper isn't renamed that way, but still has references to opentelemetry-api classes renamed. Now when you are using it from the test you'll end up using the class from the agent where the types of the arguments have been shaded.

[huange7]

when I normally run the application with Java agent, there are no errors, but the problem occurs only when running this :instrumentation:test.

[laurit]

if your application tried to used the instrumentation api that you have customized inside the application you'd probably get the same exception

[huange7]

How should I call Instrumenter.doStart in my code? If it is as you said, how should I avoid this error?

[laurit]

either add your own classes to the same package as Instrumenter or add reloacation rule to

opentelemetry-java-instrumentation/conventions/src/main/kotlin/io.opentelemetry.instrumentation.javaagent-shadowing.gradle.kts

Line 22 in d8b482a

relocate("io.opentelemetry.instrumentation", "io.opentelemetry.javaagent.shaded.instrumentation") {

[huange7]

if your application tried to used the instrumentation api that you have customized inside the application you'd probably get the same exception

I tried using Instrumenter.start in my application, and the same thing happened. Where should I add the relocate part?

[huange7]

Hi, @laurit . i add relocation rule to

opentelemetry-java-instrumentation/conventions/src/main/kotlin/io.opentelemetry.instrumentation.javaagent-shadowing.gradle.kts

Line 22 in d8b482a

relocate("io.opentelemetry.instrumentation", "io.opentelemetry.javaagent.shaded.instrumentation") {

like

  if (project.findProperty("disableShadowRelocate") != "true") {
    // prevents conflict with library instrumentation, since these classes live in the bootstrap class loader
    relocate("io.opentelemetry.instrumentation", "io.opentelemetry.javaagent.shaded.instrumentation") {
      // Exclude resource providers since they live in the agent class loader
      exclude("io.opentelemetry.instrumentation.resources.*")
      exclude("io.opentelemetry.instrumentation.spring.resources.*")
    }

    // relocate(OpenTelemetry API) since these classes live in the bootstrap class loader
    relocate("io.opentelemetry.api", "io.opentelemetry.javaagent.shaded.io.opentelemetry.api")
    relocate("io.opentelemetry.semconv", "io.opentelemetry.javaagent.shaded.io.opentelemetry.semconv")
    relocate("io.opentelemetry.context", "io.opentelemetry.javaagent.shaded.io.opentelemetry.context")
    relocate("io.opentelemetry.extension.incubator", "io.opentelemetry.javaagent.shaded.io.opentelemetry.extension.incubator")

    // my code
    relocate("io.opentelemetry.myproject.instrumentation", "io.opentelemetry.myproject.javaagent.shaded.instrumentation")
    relocate("io.opentelemetry.extension.myproject.api", "io.opentelemetry.extension.myproject.javaagent.shaded.io.opentelemetry.api")
  }

When running ./gradlew instrumentation:test, it indeed does not result in a NoSuchMethodError

To better understand the project codes and solve the problem, I have one questions:

• Why does adding a new relocation rule solve the problem?

I am grateful for any response, thanks

[laurit]

If you have the same class in the application and at the root of the agent then if you application uses parent first class loading it will load the class from the agent. Since agent build process relocates opentelemetry api classes the class in the agent is not the same as what you have in the application. The methods in the agent class have different parameter types (they were renamed) so calling a method there as you'd be calling in the version that you have inside the application will result in a NoSuchMethodError as the method does not exist. Adding a relocation rule that renames your class fixes the issue as now agent has a class with different name from what you have in the application.

[huange7]

oh, i think i understand now. before I add relocation rule, both the application and the agent were loading the same class, and the original class's parameters related to opentelemetry-api have already been relocated. Therefore, the class that the application loaded does not contain the methods of the original opentelemetry-api, but rather the methods after relocation, right?

[laurit]

yes

[huange7]

Thanks! @laurit