SIGSEGV segfault crash with PHP 8.1.29

[archon810]

Description

Hi,

We recently updated to OpenSUSE 15.6 (from 15.4) and are testing an update from PHP 8.0 to 8.1 (8.1.29 currently). On the test machine where we're testing, all of a sudden I'm seeing constant PHP8 crashes. After using gdb to see what's going on, they all seem to point to libpcre2-8-0, specifically pcre2_jit_match_8, php_pcre_replace_impl, php_pcre_replace, and php_replace_in_subject.

As has been suggested in other places like PCRE2Project/pcre2#57, oerdnj/deb.sury.org#1721, https://bugs.php.net/bug.php?id=81647, it seems like JIT is the culprit once again (the only other recent PHP crashes I've observed were related to opcache.jit being enabled).

Here's a sample backtrace from one such crash.

gdb php8 core.php.3000.pylon.1722366712
GNU gdb (GDB; devel:gcc) 14.2
...
Reading symbols from php8...
Reading symbols from /usr/lib/debug/usr/bin/php-8.1.29-lp156.2.1.x86_64.debug...
warning: Can't open file /var/run/nscd/dbUKm9bO (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libltdl.so.7.3.1 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libhogweed.so.6.9 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libnettle.so.8.9 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libcurl.so.4.8.0 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libpcre2-8.so.0.12.0 (deleted) during file-backed mapping note processing
[New LWP 3000]
warning: .dynamic section for "/usr/lib64/libpcre2-8.so.0" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libcurl.so.4" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libxcb.so.1" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libonig.so.4" is not at the expected address (wrong library or version mismatch?)
Missing separate debuginfo for /usr/lib64/libicuio.so.73.
The debuginfo package for this file is probably broken.
bMissing separate debuginfo for /usr/lib64/libicui18n.so.73.
The debuginfo package for this file is probably broken.
Missing separate debuginfo for /usr/lib64/libicuuc.so.73.
The debuginfo package for this file is probably broken.
Missing separate debuginfo for /usr/lib64/libicudata.so.73.
The debuginfo package for this file is probably broken.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/usr/bin/php <SNIPPED BY archon810>'.
Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x000073d0084ecfc5 in pcre2_jit_match_8 (code=0x5bfa81d18570, subject=subject@entry=0x73cffd616d60 "SELECT 1", length=length@entry=8, start_offset=start_offset@entry=0, options=0, options@entry=1073741824, match_data=match_data@entry=0x5bfa81db53c0, mcontext=0x5bfa81db5350) at src/pcre2_jit_match.c:142
142       arguments.offset_limit = mcontext->offset_limit;
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.38-150600.14.5.1.x86_64 libcom_err2-debuginfo-1.47.0-150600.4.3.2.x86_64 libgcc_s1-debuginfo-13.3.0+git8781-150000.1.12.1.x86_64 libgcrypt20-debuginfo-1.10.3-150600.1.23.x86_64 libglib-2_0-0-debuginfo-2.78.6-150600.4.3.1.x86_64 libgpg-error0-debu
ginfo-1.47-150600.1.3.x86_64 libjitterentropy3-debuginfo-3.4.1-150000.1.12.1.x86_64 libltdl7-debuginfo-2.4.6-150000.3.8.1.x86_64 libmemcachedprotocol0-debuginfo-1.1.4-150600.2.2.x86_64 libmemcachedutil2-debuginfo-1.1.4-150600.2.2.x86_64 libonig4-debuginfo-6.7.0-150000.3.6.1.x86_64 libopenssl3-debuginfo-3.1.4-150
600.5.10.1.x86_64 libsodium23-debuginfo-1.0.18-150000.4.6.1.x86_64 libssh4-debuginfo-0.10.6-lp156.95.1.x86_64 libtiff5-debuginfo-4.0.9-150000.45.41.1.x86_64 libwebp7-debuginfo-1.0.3-150200.3.10.1.x86_64 libxcb1-debuginfo-1.13-150000.3.11.1.x86_64 libxml2-2-debuginfo-2.10.3-150500.5.17.1.x86_64 libxslt1-debuginfo
-1.1.39-lp156.146.1.x86_64 libzip5-debuginfo-1.10.1-150600.1.5.x86_64 php8-gd-debuginfo-8.1.29-lp156.2.1.x86_64 php8-intl-debuginfo-8.1.29-lp156.2.1.x86_64 php8-openssl-debuginfo-8.1.29-lp156.2.1.x86_64 php8-sockets-debuginfo-8.1.29-lp156.2.1.x86_64 php8-tokenizer-debuginfo-8.1.29-lp156.2.1.x86_64
(gdb) bt
#0  0x000073d0084ecfc5 in pcre2_jit_match_8 (code=0x5bfa81d18570, subject=subject@entry=0x73cffd616d60 "SELECT 1", length=length@entry=8, start_offset=start_offset@entry=0, options=0, options@entry=1073741824, match_data=match_data@entry=0x5bfa81db53c0, mcontext=0x5bfa81db5350) at src/pcre2_jit_match.c:142
#1  0x00005bfa816557b2 in php_pcre_replace_impl (pce=pce@entry=0x73cfff7b2040, subject_str=subject_str@entry=0x73cffd616d48, subject=subject@entry=0x73cffd616d60 "SELECT 1", subject_len=subject_len@entry=8, replace_str=replace_str@entry=0x73cfff97f360, limit=limit@entry=18446744073709551615,
    replace_count=0x7ffcac255208) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:1664
#2  0x00005bfa816560ce in php_pcre_replace (regex=<optimized out>, subject_str=0x73cffd616d48, subject=0x73cffd616d60 "SELECT 1", subject_len=8, replace_str=0x73cfff97f360, limit=18446744073709551615, replace_count=0x7ffcac255208) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:1606
#3  0x00005bfa81656acd in php_replace_in_subject (replace_count=0x7ffcac255208, limit=<optimized out>, subject=<optimized out>, replace_ht=0x0, replace_str=<optimized out>, regex_ht=<optimized out>, regex_str=<optimized out>) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2176
#4  preg_replace_common (is_filter=false, return_value=0x73cfe548d040, execute_data=0x73cfe548d110) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2314
#5  zif_preg_replace (execute_data=0x73cfe548d110, return_value=0x73cfe548d040) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2371
#6  0x000073d007de303d in xdebug_execute_internal (current_execute_data=0x73cfe548d110, return_value=0x73cfe548d040) at /tmp/pear/temp/xdebug/src/base/base.c:1024
#7  0x00005bfa816164ae in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1981
#8  0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#9  0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#10 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#11 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#12 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#13 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#14 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#15 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#16 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#17 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#18 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#19 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#20 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
...
this repeats for tens of thousands of lines - probably an endless loop

As you can see, it looks like the query was just SELECT 1 and whatever regex was used to parse it crashes the whole PHP.

I think in our code (using Wordpress), it happens here:

            //Specify a dummy query
            if ( $wpdb->db_connect( 'SELECT 1' ) ) {
                error_reporting( $error_reporting );

                $no_database = false;
                break;
            }

From what I can tell based on the messages in /var/log/messages, the pcre library loaded is libpcre2-8.so.0.13.0, which according to the package manager is version 10.44 (the latest available).

2024-08-01T11:02:55.361543-07:00 pylon kernel: [3516751.265471] php[19527]: segfault at 7fff87da0f98 ip 0000622dafa528ad sp 00007fff87da0fa0 error 6 in php[622daf600000+7cf000] likely on CPU 1 (core 1, socket 0)
2024-08-01T11:02:57.324535-07:00 pylon kernel: [3516753.226213] php[19707]: segfault at 7ffcf0b1cff8 ip 00007169714d1fe9 sp 00007ffcf0b1d000 error 6 in libpcre2-8.so.0.13.0[71697146e000+bf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:02:58.622204-07:00 pylon kernel: [3516754.517316] php[19534]: segfault at 7ffca2843fe8 ip 0000792ef127f422 sp 00007ffca2843fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:02:59.577390-07:00 pylon kernel: [3516755.481372] php[19614]: segfault at 7ffdc5062ff0 ip 00007b13d6ed8068 sp 00007ffdc5062fd0 error 6 likely on CPU 1 (core 1, socket 0)
2024-08-01T11:03:04.225560-07:00 pylon kernel: [3516760.127703] php[19524]: segfault at 7ffd6454ffd8 ip 00007d8ab5673f12 sp 00007ffd6454ffa0 error 6 likely on CPU 1 (core 1, socket 0)
2024-08-01T11:03:07.152392-07:00 pylon kernel: [3516763.054547] php[19706]: segfault at 7ffd871d6fe0 ip 000079beaa22c068 sp 00007ffd871d6fc0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:08.288678-07:00 pylon kernel: [3516764.192171] php[19705]: segfault at 7ffdc1141ff8 ip 0000799f3c373fe9 sp 00007ffdc1142000 error 6 in libpcre2-8.so.0.13.0[799f3c310000+bf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:03:09.114457-07:00 pylon kernel: [3516765.016904] php[19523]: segfault at 7fff5910afd8 ip 00007587ea441fdf sp 00007fff5910afa0 error 6 likely on CPU 4 (core 4, socket 0)
2024-08-01T11:03:14.294606-07:00 pylon kernel: [3516770.197875] php[19532]: segfault at 7ffeccc46fd8 ip 000063fb26c528ad sp 00007ffeccc46fe0 error 6 in php[63fb26800000+7cf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:03:20.843902-07:00 pylon kernel: [3516776.745451] php[19525]: segfault at 7ffc3ae71fe8 ip 00007a2cb72c2fdf sp 00007ffc3ae71fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:24.383890-07:00 pylon kernel: [3516780.285203] php[19533]: segfault at 7ffdfb842fe8 ip 000079fb5177dfdf sp 00007ffdfb842fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:27.010622-07:00 pylon kernel: [3516782.913697] php[19704]: segfault at 7fff0dff3ff8 ip 00007d753d687fe9 sp 00007fff0dff4000 error 6 in libpcre2-8.so.0.13.0[7d753d624000+bf000] likely on CPU 4 (core 4, socket 0)

zypper info libpcre2-8-0
Loading repository data...
Reading installed packages...


Information for package libpcre2-8-0:
-------------------------------------
Repository     : openSUSE BuildService - devel:libraries:c_c++
Name           : libpcre2-8-0
Version        : 10.44-lp156.80.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:libraries:c_c++
Installed Size : 973.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : pcre2-10.44-lp156.80.1.src
Upstream URL   : https://www.pcre.org
Summary        : A library for Perl-compatible regular expressions
Description    : 
    The PCRE2 library is a set of functions that implement regular
    expression pattern matching using the same syntax and semantics
    as Perl 5.

    PCRE2 is a re-working of the original PCRE library to provide an entirely new
    API.

    This PCRE2 library variant supports 8-bit and UTF-8 strings.
    (See also libpcre2-16 and libpcre2-32)

I'm posting it here in case it's an issue with PHP itself.

I also posted to PCRE2Project/pcre2#435. Tried disabling pcre.jit but it didn't work.

PHP Version

PHP 8.1.29

Operating System

OpenSUSE 15.6

[devnexen]

warning: .dynamic section for "/usr/lib64/libpcre2-8.so.0" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libcurl.so.4" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libxcb.so.1" is not at the expected address (wrong library or version mismatch?)
...

Quick question, with which opensuse release 8.1.29 comes from/had been built ?

The Leap 15.6 release incorporates several key software upgrades enhancing performance and security. It integrates Linux Kernel 6.4, which provides backports for some of latest hardware drivers, which offer performance enhancements. OpenSSL 3.1 becomes the new default and provides robust security features and updated cryptographic algorithms. Database management systems receive significant updates with MariaDB 10.11.6 and PostgreSQL 16. Redis 7.2 offers advanced data handling capabilities and the software stack is rounded out with PHP 8.2

[archon810]

php8│ │8.1.29-lp156.2.1 │openSUSE BuildService - devel:languages:php

In this case, it came from this repo:

zypper addrepo -f  --name "openSUSE BuildService - devel:languages:php"               https://download.opensuse.org/repositories/devel:/languages:/php:/php81/openSUSE_Leap_15.6/ "download.opensuse.org-php"

We also tested PHP 8.2, but some Wordpress plugins we're using haven't been updated for it yet and output a lot of notices/warnings, so the plan was to move from 8.0 to 8.1 for now.

[cmb69]

First, PHP 8.1 is in security mode now, so won't receive any regular bug fixes, but only security relevant fixes. This doesn't look like a security issue, though.

We also tested PHP 8.2, but some Wordpress plugins we're using haven't been updated for it yet and output a lot of notices/warnings, so the plan was to move from 8.0 to 8.1 for now.

That is unfortunate. Maybe you can urge the respective developers a bit to make their code compatible with PHP 8.2, and then switch to that version directly.

Tried disabling pcre.jit but it didn't work.

That would be an indication that PCRE's JIT is not the problem.

Anyhow, did you try running without Xdebug? Would that also crash?

[github-actions]

No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you.

[archon810]

Anyhow, did you try running without Xdebug? Would that also crash?

Yes, it was also crashing without xdebug. All signs pointed to pcre.jit, so I'm stumped.

That is unfortunate. Maybe you can urge the respective developers a bit to make their code compatible with PHP 8.2, and then switch to that version directly.

We have been. Right now the biggest offender is google/site-kit-wp#8331, though they have been closing some tickets about notices. I don't believe it's fully ready yet though.

[cmb69]

From PCRE2Project/pcre2#435:

Here's how it's set in both /etc/php8/apache/php.ini and /etc/php8/cli/php.ini:

pcre.jit=0

I verified that the value is set correctly:

php8 -r 'echo phpinfo();'|ack pcre.jit
pcre.jit => 0 => 0

That verification is irrelevant; you need to check when running a web request through Apache. Put a file containing

<?php
phpinfo();

somewhere in the docroot, and request that from a browser. I have the suspicion that neither disabling pcre.jit nor disabling Xdebug actually worked in your experiments so far.