Skip to content

Conversation

@laurit
Copy link
Contributor

@laurit laurit commented Mar 27, 2025

Fixes indy test failures in #13590
These failures are caused by circularity for classes in java.lang.invoke. Ignoring that package shouldn't be an issue because the purpose of the early instrumentations is to set up virtual field transforms for executors instrumentation as early as possible. Since virtual field instrumentation does structural modifications to classes it can not be applied to already loaded classes. The classes that we wish to apply these transforms to aren't in the java.lang.invoke package.

@laurit laurit requested a review from a team as a code owner March 27, 2025 11:58
agentBuilder
.ignore(
target -> instrumentationInstalled, // turn off after instrumentation is installed
Objects::nonNull)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the Objects::nonNull(checks that instrumentation applies only in boot loader) because it doesn't work as intended. When both type and class loader matcher are given both must match for the type to be ignored, as far as I understand here the intent was to ignore when either matches. If we want we can add the class loader check with .or(Objects::nonNull) but it shouldn't be necessary.

@trask
Copy link
Member

trask commented Mar 27, 2025

cc @JonasKunz

Copy link
Contributor

@JonasKunz JonasKunz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though I would be curious about the original error/exception that surfaced in #13590, I haven't been able to find it.

Maybe we should also add the flag to test with OpenJ9 to #13590, just to make sure there are no other things going wrong there?

@laurit
Copy link
Contributor Author

laurit commented Mar 27, 2025

With java11 it gets the following exception

[otel.javaagent 2025-03-27 17:48:22:878 +0200] [main] DEBUG io.opentelemetry.javaagent.bootstrap.ExceptionLogger - Error bootstrapping indy instruction
java.lang.ClassCircularityError: java/lang/invoke/DirectMethodHandle$EnsureInitialized
	at java.base/java.lang.invoke.DirectMethodHandle.checkInitialized(DirectMethodHandle.java:378)
	at java.base/java.lang.invoke.DirectMethodHandle.ensureInitialized(DirectMethodHandle.java:368)
	at java.base/java.lang.invoke.DirectMethodHandle.internalMemberNameEnsureInit(DirectMethodHandle.java:318)
	at io.opentelemetry.javaagent.bootstrap.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:43)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:138)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:258)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:248)
	at java.base/java.lang.ClassLoader.getResource(ClassLoader.java:1408)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader$BootstrapClassLoaderProxy.getResource(AgentClassLoader.java:373)
	at java.base/java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1742)
	at net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:513)
	at net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:494)
	at net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:2127)
	at net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:895)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool.doResolve(AgentCachingPoolStrategy.java:352)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool$CachingTypeDescription.delegate(AgentCachingPoolStrategy.java:422)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool$LazyTypeDescription.getInterfaces(AgentCachingPoolStrategy.java:503)
	at net.bytebuddy.description.type.TypeDescription$Generic$OfNonGenericType.getInterfaces(TypeDescription.java:3758)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher$SafeInterfaceIterator.<init>(SafeHasSuperTypeMatcher.java:166)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher$SafeInterfaceIterator.<init>(SafeHasSuperTypeMatcher.java:156)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.safeGetInterfaces(SafeHasSuperTypeMatcher.java:100)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.hasInterface(SafeHasSuperTypeMatcher.java:86)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.matches(SafeHasSuperTypeMatcher.java:69)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.matches(SafeHasSuperTypeMatcher.java:39)
	at net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:148)
	at io.opentelemetry.javaagent.tooling.util.IgnoreFailedTypeMatcher.matches(IgnoreFailedTypeMatcher.java:27)
	at io.opentelemetry.javaagent.tooling.util.IgnoreFailedTypeMatcher.matches(IgnoreFailedTypeMatcher.java:18)
	at io.opentelemetry.javaagent.tooling.util.NamedMatcher.matches(NamedMatcher.java:26)
	at net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1965)
	at net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$Conjunction.matches(AgentBuilder.java:1800)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12702)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12660)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12369)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13151)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13081)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12603)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.invoke.DirectMethodHandle.checkInitialized(DirectMethodHandle.java:378)
	at java.base/java.lang.invoke.DirectMethodHandle.ensureInitialized(DirectMethodHandle.java:368)
	at java.base/java.lang.invoke.DirectMethodHandle.internalMemberNameEnsureInit(DirectMethodHandle.java:318)
	at io.opentelemetry.javaagent.bootstrap.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:43)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:138)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:258)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:248)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1020)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.defineClass(AgentClassLoader.java:189)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.findAgentClass(AgentClassLoader.java:182)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.loadClass(AgentClassLoader.java:157)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:207)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:110)
	at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

tests pass but we detect that there exceptions were logged (usually indicates advice failure) and fail the tests because of that. On jdk 21 there is a similar exception

[otel.javaagent 2025-03-27 17:50:02:082 +0200] [main] DEBUG io.opentelemetry.javaagent.bootstrap.ExceptionLogger - Error bootstrapping indy instruction
java.lang.ClassCircularityError: java/lang/invoke/DirectMethodHandle$1
	at java.base/java.lang.invoke.DirectMethodHandle.ensureInitialized(DirectMethodHandle.java:373)
	at java.base/java.lang.invoke.DirectMethodHandle.internalMemberNameEnsureInit(DirectMethodHandle.java:336)
	at io.opentelemetry.javaagent.bootstrap.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:43)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:147)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
	at java.base/java.lang.ClassLoader.getResource(ClassLoader.java:1420)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader$BootstrapClassLoaderProxy.getResource(AgentClassLoader.java:373)
	at java.base/java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1762)
	at net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:513)
	at net.bytebuddy.dynamic.ClassFileLocator$ForClassLoader.locate(ClassFileLocator.java:494)
	at net.bytebuddy.dynamic.ClassFileLocator$Compound.locate(ClassFileLocator.java:2127)
	at net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:895)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool.doResolve(AgentCachingPoolStrategy.java:352)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool$CachingTypeDescription.delegate(AgentCachingPoolStrategy.java:422)
	at io.opentelemetry.javaagent.tooling.muzzle.AgentCachingPoolStrategy$AgentTypePool$LazyTypeDescription.getInterfaces(AgentCachingPoolStrategy.java:503)
	at net.bytebuddy.description.type.TypeDescription$Generic$OfNonGenericType.getInterfaces(TypeDescription.java:3758)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher$SafeInterfaceIterator.<init>(SafeHasSuperTypeMatcher.java:166)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher$SafeInterfaceIterator.<init>(SafeHasSuperTypeMatcher.java:156)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.safeGetInterfaces(SafeHasSuperTypeMatcher.java:100)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.hasInterface(SafeHasSuperTypeMatcher.java:86)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.hasInterface(SafeHasSuperTypeMatcher.java:91)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.matches(SafeHasSuperTypeMatcher.java:69)
	at io.opentelemetry.javaagent.extension.matcher.SafeHasSuperTypeMatcher.matches(SafeHasSuperTypeMatcher.java:39)
	at net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:148)
	at io.opentelemetry.javaagent.tooling.util.IgnoreFailedTypeMatcher.matches(IgnoreFailedTypeMatcher.java:27)
	at io.opentelemetry.javaagent.tooling.util.IgnoreFailedTypeMatcher.matches(IgnoreFailedTypeMatcher.java:18)
	at io.opentelemetry.javaagent.tooling.util.NamedMatcher.matches(NamedMatcher.java:26)
	at net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1965)
	at net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$Conjunction.matches(AgentBuilder.java:1800)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12702)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12660)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12369)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13151)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:13081)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12603)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:610)
	at java.base/java.lang.invoke.DirectMethodHandle.ensureInitialized(DirectMethodHandle.java:373)
	at java.base/java.lang.invoke.DirectMethodHandle.internalMemberNameEnsureInit(DirectMethodHandle.java:336)
	at io.opentelemetry.javaagent.bootstrap.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:43)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:147)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:274)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:264)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1025)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.defineClass(AgentClassLoader.java:189)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.findAgentClass(AgentClassLoader.java:182)
	at io.opentelemetry.javaagent.bootstrap.AgentClassLoader.loadClass(AgentClassLoader.java:157)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:207)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:110)
	at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:560)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:572)

and also the following exception

java.lang.ExceptionInInitializerError
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
	at org.gradle.process.internal.worker.messaging.WorkerConfigSerializer.<clinit>(WorkerConfigSerializer.java:46)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:86)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.NullPointerException: Cannot read the array length because "<local5>" is null
	at java.base/java.lang.Class.privateGetPublicMethods(Class.java:3614)
	at java.base/java.lang.Class.getMethods(Class.java:2185)
	at java.base/java.lang.reflect.Proxy$ProxyBuilder.referencedTypes(Proxy.java:730)
	at java.base/java.lang.reflect.Proxy$ProxyBuilder.<init>(Proxy.java:632)
	at java.base/java.lang.reflect.Proxy$ProxyBuilder.<init>(Proxy.java:643)
	at java.base/java.lang.reflect.Proxy.lambda$getProxyConstructor$0(Proxy.java:429)
	at java.base/jdk.internal.loader.AbstractClassLoaderValue$Memoizer.get(AbstractClassLoaderValue.java:329)
	at java.base/jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent(AbstractClassLoaderValue.java:205)
	at java.base/java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:427)
	at java.base/java.lang.reflect.Proxy.newProxyInstance(Proxy.java:1034)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter.<init>(ProxyDispatchAdapter.java:49)
	at org.gradle.internal.event.ListenerBroadcast.getSource(ListenerBroadcast.java:59)
	at org.gradle.internal.logging.sink.OutputEventRenderer.<init>(OutputEventRenderer.java:71)
	at org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext.reset(OutputEventListenerBackedLoggerContext.java:89)
	at org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext.<init>(OutputEventListenerBackedLoggerContext.java:48)
	at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:30)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:27)
	... 10 more

that kills the test.

@JonasKunz
Copy link
Contributor

JonasKunz commented Mar 27, 2025

Thanks! So based on the last method being invoked before indy bootstrap (ClassLoader.getResource(ClassLoader.java:1420)) my guess is that the ResourceInjectionInstrumentation is the offender. We should maybe consider forcing inlining for that instrumentation like I did for BootDelegationInstrumentation, at least if we run into this again.

EDIT: The DefineClassInstrumentation is involved too: java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1025)

If you want to, I can take care of that.

@laurit laurit merged commit 5626e25 into open-telemetry:main Mar 27, 2025
86 checks passed
@laurit laurit deleted the ignore-invoke branch March 27, 2025 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants