NPE during JIT regex compilation on Corretto-21.0.11.10.1

[wmccall-amplify]

Describe the bug

We ran into a SIGSEGV error which caused the JVM to crash and export a hs_err_pid<pid>.log file. The log reported the following:

  # A fatal error has been detected by the Java Runtime Environment:
  #  SIGSEGV (0xb) at pc=..., pid=..., tid=...
  # Problematic frame:
  # J ... c2 java.util.regex.Pattern.RemoveQEQuoting()V java.base@21.0.11 (370 bytes) @ 0x... [0x...+0x...]
  si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Which seems to point to a null pointer dereference inside C2 JIT-compiled java.util.regex.Pattern.RemoveQEQuoting(). This caused downtime of our auth service in a production environment, negatively impacting many real users.

To Reproduce

Unfortunately we are unable to reproduce. We wrote a script to force the JVM to go through this path millions of times in an attempt to cause the error, but we were unsuccessful.

Expected behavior

JIT compiler successfully compiles regex and proceeds.

Screenshots

N/A

Platform information

We are running the open source auth service Keycloak v26.3.5 on EC2 instances with the following environment:

• JRE: OpenJDK Runtime Environment Corretto-21.0.11.10.1 (21.0.11+10-LTS)
• OS: Amazon Linux 2023
• Uptime before crash: ~2.5 days

Additional context

• Pattern being compiled at crash time: a single \x0A (newline) character — patternLength = 1
• Call chain on stack: Matcher::replaceAll → Pattern::compile → RemoveQEQuoting
• Crash occurred ~immediately after C2 JIT-compiled RemoveQEQuoting (compile timestamp matches crash timestamp within milliseconds), suggesting a C2 miscompilation rather than a pre-existing problem in the compiled code
• Currently attempting to work around this issue with the java option -XX:CompileCommand=exclude,java.util.regex.Pattern::RemoveQEQuoting

redacted_hs_err_pid8906.log

[simonis]

Hi @wmccall-amplify,

Thanks for this reports. I don't think that the "Crash occurred ~immediately after C2 JIT-compiled RemoveQEQuoting"? I can't see RemoveQEQuoting() in the Compilation events section of the hs_err file. The timestamp that gets printed when the nmethod is output in the Stack slot to memory mapping section is the current time when the printing occurs. Instead, you have to look at the compilation id (34025 for RemoveQEQuoting()) which is much lower than the latest compilation id 77676 in the Compilation events section. This means that RemoveQEQuoting() was compiled quite some time before the crash and it was very likely used many times before the crash as well without any issue.

The strange thing about the crash is that it happens right at the beginning of the method (i.e. the first instruction in the Verified Entry Point) which does the stack banging:

[Entry Point]
   0x7f175587ca20:  mov r10d, dword ptr [rsi + 8]             ; Loads compressed class pointer from object header
   0x7f175587ca24:  movabs r11, 0x7ef029000000                ; Loads class base address
   0x7f175587ca2e:  add r10, r11                              ; Decompresses the pointer by adding base address
   0x7f175587ca31:  cmp rax, r10                              ; Inline cache check (compares class with expected value)
   0x7f175587ca34:  jne 0x7f1753c1d280                        ; On miss: jumps to runtime Inline Cache stub
   0x7f175587ca3a:  nop                                       ; 2-byte alignment padding
   0x7f175587ca3c:  nop dword ptr [rax]                       ; 4-byte alignment padding
[Verified Entry Point]
=> 0x7f175587ca40:  mov dword ptr [rsp - 0x14000], eax        ; Stack bang (JVM stack overflow check)
   0x7f175587ca47:  push rbp                                  ; Saves the previous frame pointer
   0x7f175587ca48:  sub rsp, 0x50                             ; Allocates 80 bytes for the stack frame

Taking into account that the stack of the current thread is at [0x00007eea67900000,0x00007eea67a00000] with the stack pointer being 0x00007eea679fe2e8, I don't see how the stack bang at 0x00007eea679fe2e8 - 0x14000 = 0x7eea679ea2e8 can lead to a SIGSEGV, because it points right into the middle of the threads stack area. Also, the si_addr value of 0x0000000000000000 in the siginfo structure doesn't correspond to this address.

The mixed stack trace at the beginning of the hs_err file is corrupted right after the first frame probably because the crash in RemoveQEQuoting() happened right at the first instruction, before the stack frame was set up. But I agree with your analysis of the most probable call chain which lead to the crash (i.e. Matcher::replaceAll() → Pattern::compile() → RemoveQEQuoting()) which can be derived from the nmethod pointers in the Stack slot to memory mapping section.

All together, the crash looks very mysterious and I'm afraid there isn't much we can do for know until we get any additional data or evidence. I see that you are using the DataDog agent which does class redefinition and transformation and JFR which might be the reason for the crash, but these are currently really very vague guesses. However, I don't think you have to exclude RemoveQEQuoting() from compilation because I don't see any indication that it is the reason for the crash.

[jichen-amplify]

Hi @simonis, thanks so much for shedding some light on this issue. I am in the same team with @wmccall-amplify. Since it seems that we would need additional data to find out the root cause for this, could you let us know if there are anything we could have captured during the crash that would have helped with the investigation? We were also wondering if you have seen anything like this before in other reports, maybe to evaluate the likelihood of such event happening again. Thanks again for your help!

[simonis]

Hi @jichen-amplify, as I wrote, the error looks very mysterious and I have no explanation how and why it happened. Capturing and providing the hs_err file is very useful you should always do that. Apart from that, I think the only thing that could help in this case would be a test case which can reproduce the crash but I think that would be hard for this specific issue.

My personal feeling is, that the likelihood for this specific error is minimal. The only recommendation I can give you so far is to monitor your servers for hs_error files. This will give you a good estimation on how frequently the JVM crashes (ideally you won't find any :).

Please feel free to report any additional crashes here.