Issue when run opentelemetry agent in webshpere application server (Dump Event "gpf" (00002000) received )

[DuongChien1812]

Describe the bug

When run opentelemetry agent in webshpere application server i got Dump Event "gpf" (00002000) received and after that server was crash
this is follow config in application server:
JVM setting
-javaagent:/opentelemetry-javaagent.jar
Param:
OTEL_EXPORTER_OTLP_ENDPOINT: xxxxxxxxx

And I diagnostic javacore and dump file, I found this :
Module Issue:
/u01/IBM/WebSphere/AppServer/java/8.0/jre/lib/amd64/compressedrefs/libj9jit29.so
Method compiling:
io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.setTypeMatcher(...)
Error address: InaccessibleAddress: 0x0 → Null Pointer dereference in native code

Steps to reproduce

Delete param agent instrument

Expected behavior

Can run withthuot issue and crash server

Actual behavior

Module Issue:
/u01/IBM/WebSphere/AppServer/java/8.0/jre/lib/amd64/compressedrefs/libj9jit29.so
Method compiling:
io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.setTypeMatcher(...)
Error address: InaccessibleAddress: 0x0 → Null Pointer dereference in native code

Javaagent or library instrumentation version

v2.18.1

Environment

OS: Linux kernel 3.10.0-862.el7.x86_64 (CentOS/RHEL 7)
CPU: 12 core, amd64
JVM: IBM J9 8.0.5.26 (SR5 FP26, build 20181115)
Agent: opentelemetry-javaagent.jar (lastest version: 2.18.1 )
JDK: 8.0_64

Additional context

0SECTION TITLE subcomponent dump routine
NULL ===============================
1TICHARSET UTF-8
1TISIGINFO Dump Event "gpf" (00002000) received
1TIDATETIME Date: 2025/08/07 at 22:31:44:740
1TINANOTIME System nanotime: 11324915772283202
1TIFILENAME Javacore filename: /u01/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20250807.223136.54505.0002.txt
1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x100 (trace_disabled)
1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[davisjose19]

Hi @DuongChien1812 -

Take a look at 14398 - Seems like we are both facing a similar issue?

[laurit]

To me this looks like a bug in the ibm jvm. Since it says

Method compiling:
io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.setTypeMatcher(...)

We can assume that something in that method breaks the jit complier. You could try excluding that method from the jit compilation as described in https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xjit-xnojit#exclude Or switch to a more recent ibm vm. We run tests with recent openj9 and have not seen such crashes. Alternatively you can work with IBM support.
Often such crashes can be averted by doing minor changes to the problematic method, such as rearranging it a bit or splitting it. We are open to a PR that modifies the method in a way that prevents this crash.

[davisjose19]

@laurit - Thanks for this suggestion. Won't this (excluding JIT) degrade performance to an extend?

[laurit]

@laurit - Thanks for this suggestion. Won't this (excluding JIT) degrade performance to an extend?

Disabling jit altogether definitely will hurt performance. In this case you would only need to exclude one problematic method from jit complication. Since this method is only called during agent startup it shouldn't have a noticeable impact.

[davisjose19]

@laurit - Thanks for this suggestion. Won't this (excluding JIT) degrade performance to an extend?

Disabling jit altogether definitely will hurt performance. In this case you would only need to exclude one problematic method from jit complication. Since this method is only called during agent startup it shouldn't have a noticeable impact.

Will try this and let you know.

[DuongChien1812]

Hi @davisjose19 ,
I was try click on Disable JIT in Process definition> Java Virtual Machine and it didn't have that issue above. Follow instruction, disable will impact to perform (jit complication) but it will be fine use opentelemetry with troubleshoot and debug. So you turn it off when don't need troubleshoot.

[davisjose19]

Hi @laurit / @DuongChien1812 -
I am unable to exclude JIT completely. So i resorted to exclusion of the problematic method -

-Xjit:exclude={io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.setTypeMatcher()*}

On starting the servers - I am getting the thread dump errors and crash for another method :

Unhandled exception
Type=Segmentation error vmState=0x00050080
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F707CA9ABE0 Handler2=00007F707C382030 InaccessibleAddress=0000000000000000
RDI=00007F6FB3021300 RSI=0000000000000002 RAX=0000000000000001 RBX=0000000000000001
RCX=00007F6FB3070B40 RDX=00007F7073297E00 R8=0000000000000000 R9=0000000000000001
R10=0000000000000002 R11=0000000000000000 R12=0000000000000000 R13=0000000000000002
R14=00007F6FB3021300 R15=00007F7077347820
RIP=00007F707684844A GS=0000 FS=0000 RSP=00007F70732958D0
EFlags=0000000000010213 CS=0033 RBP=0000000000000000 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 00000000ff000000 (f: 4278190080.000000, d: 2.113707e-314)
xmm1 2525252525252525 (f: 623191360.000000, d: 9.532824e-130)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 ff00000000000000 (f: 0.000000, d: -5.486124e+303)
xmm4 00000000003b7463 (f: 3896419.000000, d: 1.925087e-317)
xmm5 2f6d62692f6d6f63 (f: 795701120.000000, d: 3.097763e-80)
xmm6 fffffffcfffffffc (f: 4294967296.000000, d: -nan)
xmm7 00007f6fb3130820 (f: 3004368896.000000, d: 6.922735e-310)
xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm9 00007f6fb3132250 (f: 3004375552.000000, d: 6.922735e-310)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 000000004c8f5c29 (f: 1284463616.000000, d: 6.346094e-315)
xmm12 0000000049378032 (f: 1228374016.000000, d: 6.068974e-315)
xmm13 000000004a656045 (f: 1248157824.000000, d: 6.166719e-315)
xmm14 000000004b195eab (f: 1259953792.000000, d: 6.224999e-315)
xmm15 0000000049756450 (f: 1232430208.000000, d: 6.089014e-315)
Module=/opt/ssfs/runtime/jdk/jre/lib/amd64/compressedrefs/libj9jit29.so
Module_base_address=00007F70766A6000

Method_being_compiled=io/opentelemetry/javaagent/shaded/instrumentation/jdbc/internal/JdbcUtils.unwrapConnection(Ljava/sql/Connection;)Ljava/sql/Connection;

Seems like this might keep happening for more than one method ?

[laurit]

Hi @laurit / @DuongChien1812 - I am unable to exclude JIT completely. So i resorted to exclusion of the problematic method -

-Xjit:exclude={io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.setTypeMatcher()*}

Imo you have not used the exclude correctly. Shouldn't you use / as the package separator? Doesn't setTypeMatcher()* mean that you are excluding a method that takes 0 arguments, but the real method takes more?

Seems like this might keep happening for more than one method ?

It certainly is possible that there is more then one method that triggers the jvm bug. You could start with -Xjit:exclude={io/opentelemetry/javaagent/*}