Possible kernel crash on vfork

[arthur-proglove]

Crash report

What happened?

We are running a Python application on a Raspberry Pi Zero 2 W. From time to time, we are getting a kernel crash pointing at sys_vfork from the python process.

While we cannot figure out when that exactely happens and why, I found this issue regarding the use of vfork vs fork and was wondering if this could be the issue #91401.

Kernel trace:

8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 00000014 when read
[00000014] *pgd=03f2c835, *pte=00000000, *ppte=00000000
Internal error: Oops: 17 [#1] SMP ARM
Modules linked in: xt_recent usb_f_ecm u_ether usb_f_mass_storage usb_f_acm u_serial dwc2 roles cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg btnxpuart bluetooth ecdh_generic ecc crc8 moal(O) mlan(O) xt_hl cfg80211 ip6t_rt rfkill ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink binfmt_misc sc16is7xx regmap_i2c raspberrypi_hwmon i2c_bcm2835 raspberrypi_gpiomem fixed uio_pdrv_genirq uio panel_waveshare_dsi w5100_spi w5100 libcomposite i2c_dev deflate zstd ubifs ubi ofpart spi_nor mtd spi_bcm2835 drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6 overlay
CPU: 1 PID: 484 Comm: python3 Tainted: G           O       6.6.47-v7 #1
Hardware name: BCM2835
PC is at memcg_charge_kernel_stack+0xc/0x9c
LR is at copy_process+0xcc8/0x1d5c
pc : [<80119d90>]    lr : [<8011c448>]    psr: 60000013
sp : 9ff51e10  ip : 00000000  fp : 00000000
r10: 9a7d5d7c  r9 : ffffffff  r8 : 9ff51ecc
r7 : 84b1d8c0  r6 : 9ff51f48  r5 : 7ea183c8  r4 : 8100e90c
r3 : 00002a11  r2 : 00002a10  r1 : 00000000  r0 : 00000000
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 10c5383d  Table: 05c5806a  DAC: 00000055
Register r0 information: NULL pointer
Register r1 information: NULL pointer
Register r2 information: non-paged memory
Register r3 information: non-paged memory
Register r4 information: non-slab/vmalloc memory
Register r5 information: non-paged memory
Register r6 information: 2-page vmalloc region starting at 0x9ff50000 allocated at kernel_clone+0xac/0x3a8
Register r7 information: slab task_struct start 84b1d8c0 pointer offset 0 size 4544
Register r8 information: 2-page vmalloc region starting at 0x9ff50000 allocated at kernel_clone+0xac/0x3a8
Register r9 information: non-paged memory
Register r10 information: non-slab/vmalloc memory
Register r11 information: NULL pointer
Register r12 information: NULL pointer
Process python3 (pid: 484, stack limit = 0x0af81d55)
Stack: (0x9ff51e10 to 0x9ff52000)
1e00:                                     8100e90c 7ea183c8 9ff51f48 8011c448
1e20: 00000dc2 0000065f 00000000 ffffffff 8011d624 80ace5f8 9ff51e98 00004100
1e40: 821d8000 9ff51f48 00000000 00000000 00000000 00000000 8116b41c 00000000
1e60: 00000000 00000000 9a7d5d7c 005659e8 00000000 801b9424 84b1d8c0 801d1d98
1e80: 00000000 00000000 9ef8e548 00000000 012f8586 8172d880 9ff51e98 00000000
1ea0: 9ff51f00 9ff51fb0 00000000 00000000 9ff51ee0 00000008 00000000 84b1d8c0
1ec0: 00000008 802e1b80 84b1d8c0 00000000 00000000 00000000 84a89538 218c150c
1ee0: 00000000 00004100 7ea183c8 9ff51f48 000000be 00000000 84b1d8c0 000000be
1f00: 00000000 8011d624 00000001 00000000 00000000 00000000 00000000 218c150c
1f20: 00000089 00000000 7ea183c8 70ab1b40 000000be 80100298 84b1d8c0 000000be
1f40: 00000000 8011dc2c 00004100 00000000 00000000 00000000 00000000 00000000
1f60: 00000011 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1f80: 00000000 00000000 00000000 00000000 00000000 00000000 7ea183c8 218c150c
1fa0: 7fffffff 80100040 00000000 7ea183c8 72d982d8 70ab1b40 00000000 00000000
1fc0: 00000000 7ea183c8 70ab1b40 000000be 00000000 00000000 00000000 00000000
1fe0: 72d982d8 7ea1826c 00273d80 76d91320 20000010 72d982d8 00000000 00000000
 memcg_charge_kernel_stack from copy_process+0xcc8/0x1d5c
 copy_process from kernel_clone+0xac/0x3a8
 kernel_clone from sys_vfork+0x4c/0x70
 sys_vfork from ret_fast_syscall+0x0/0x4c
Exception stack(0x9ff51fa8 to 0x9ff51ff0)
1fa0:                   00000000 7ea183c8 72d982d8 70ab1b40 00000000 00000000
1fc0: 00000000 7ea183c8 70ab1b40 000000be 00000000 00000000 00000000 00000000
1fe0: 72d982d8 7ea1826c 00273d80 76d91320
Code: ea0168e2 e92d4070 e52de004 e28dd004 (e5903014) 
---[ end trace 0000000000000000 ]---

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.2 (main, May 2 2024, 11:59:00) [GCC 12.2.0]

[ZeroIntensity]

Hmm, I was under the impression that RPIs ran on MicroPython, not CPython. Could you double check that?

If you are running CPython, does this occur on 3.12? 3.11 only gets security fixes now. It would also be helpful if you posted the (Python) code that causes the problem, because I don't see any interpreter functions in the stack trace, so it isn't all that useful.

[vstinner]

We are running a Python application on a Raspberry Pi Zero 2 W. From time to time, we are getting a kernel crash pointing at sys_vfork from the python process.

Can you write a reproducer? (short code reproducing the crash)

Unable to handle kernel NULL pointer dereference at virtual address 00000014 when read

It looks like a kernel bug or a memory corruption. You might try to run your program in Valgrind.

[izbyshev]

Unable to handle kernel NULL pointer dereference at virtual address 00000014 when read

It looks like a kernel bug or a memory corruption.

Indeed, the dereference happens in the kernel, so it's a kernel bug.

You might try to run your program in Valgrind.

The problem is not in CPython, and not even in user space, so Valgrind is irrelevant here.

#91401 is also irrelevant (and is itself bogus).

If the user believes that vfork() triggers this kernel bug, they might try to reproduce with subprocess._USE_VFORK set to False.

In any case, this bug should be reported to the maintainers of the kernel used in that Raspberry Pi device, not here.