[BUG] `__vb2_queue_cancel` error in kernel logs

[goombado]

Describe the bug
In the latest version of rpicam-apps, we have been getting issues where running rpicam-still repeatedly over time inevitably and consistently causes a kernel crash – the whole Raspberry Pi 5 running the code freezes and becomes completely unresponsive (no networking etc.).

Bug report
We call the command

/usr/bin/rpicam-still --width 4624 --height 3472 --camera 0 -o /dev/shm/tmp.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0

once every 60 seconds. After ~10 hours (time varies with each iteration), we observe the following kernel logs. The errors occur 5 seconds after we request a photo to be taken on each of the two cameras plugged into the rpi.

Error Log 1: __vb2_queue_cancel

Jul 30 10:39:15 camera-testing kernel: rp1-cfe 1f00110000.csi: Using a link rate of 720 Mbps
Jul 30 10:39:15 camera-testing kernel: rp1-cfe 1f00128000.csi: Using a link rate of 720 Mbps
Jul 30 10:39:20 camera-testing kernel: Unable to handle kernel paging request at virtual address dead000000000140
Jul 30 10:39:20 camera-testing kernel: Mem abort info:
Jul 30 10:39:20 camera-testing kernel:   ESR = 0x0000000096000004
Jul 30 10:39:20 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 30 10:39:20 camera-testing kernel:   SET = 0, FnV = 0
Jul 30 10:39:20 camera-testing kernel:   EA = 0, S1PTW = 0
Jul 30 10:39:20 camera-testing kernel:   FSC = 0x04: level 0 translation fault
Jul 30 10:39:20 camera-testing kernel: Data abort info:
Jul 30 10:39:20 camera-testing kernel:   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
Jul 30 10:39:20 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Jul 30 10:39:20 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 30 10:39:20 camera-testing kernel: [dead000000000140] address between user and kernel address ranges
Jul 30 10:39:20 camera-testing kernel: Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
Jul 30 10:39:20 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc brcmfmac_wcc vc4 snd_soc_hdmi_codec ov64a40 bu64754 drm_display_helper v4l2_cci regmap_i2c brcmfmac hci_uart cec brcmutil drm_dma_helper btbcm bluetooth snd_soc_core cfg80211 aes_ce_blk aes_ce_cipher ghash_ce snd_compress gf128mul ecdh_generic rpi_hevc_dec snd_pcm_dmaengine pisp_be sha2_ce ecc rp1_cfe snd_pcm sha256_arm64 rfkill sha1_ce v4l2_mem2mem snd_timer libaes v4l2_fwnode v4l2_async videobuf2_dma_contig v3d snd videobuf2_memops sha1_generic w1_gpio videobuf2_v4l2 raspberrypi_hwmon gpu_sched wire videodev drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_fw raspberrypi_gpiomem rp1_adc rp1_mailbox nvmem_rmem uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6 spidev spi_bcm2835 i2c_brcmstb gpio_keys i2c_designware_platform i2c_designware_core
Jul 30 10:39:20 camera-testing kernel: CPU: 3 UID: 1006 PID: 97523 Comm: libcamera-still Not tainted 6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Jul 30 10:39:20 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Jul 30 10:39:20 camera-testing kernel: pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Jul 30 10:39:20 camera-testing kernel: pc : __vb2_queue_cancel+0xe0/0x2d8 [videobuf2_common]
Jul 30 10:39:20 camera-testing kernel: lr : __vb2_queue_cancel+0xa8/0x2d8 [videobuf2_common]
Jul 30 10:39:20 camera-testing kernel: sp : ffffc0008265bb50
Jul 30 10:39:20 camera-testing kernel: x29: ffffc0008265bb50 x28: ffffd06fbd8a8938 x27: ffffd06fbd7b90f8
Jul 30 10:39:20 camera-testing kernel: x26: 0000000000000000 x25: ffff800007987668 x24: ffff800007601a40
Jul 30 10:39:20 camera-testing kernel: x23: 0000000000000000 x22: ffff800007987698 x21: 0000000000000000
Jul 30 10:39:20 camera-testing kernel: x20: 0000000000000009 x19: dead000000000100 x18: 0000000000000000
Jul 30 10:39:20 camera-testing kernel: x17: 0000000000000000 x16: ffffd06fce50a858 x15: 00007ffef8023ea0
Jul 30 10:39:20 camera-testing kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
Jul 30 10:39:20 camera-testing kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fce50a8b8
Jul 30 10:39:20 camera-testing kernel: x8 : ffffc0008265bd48 x7 : 0000000000000005 x6 : 0000000000000000
Jul 30 10:39:20 camera-testing kernel: x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff80009f906d80
Jul 30 10:39:20 camera-testing kernel: x2 : 0000000000000020 x1 : ffff80000928f300 x0 : 00000000ffffffff
Jul 30 10:39:20 camera-testing kernel: Call trace:
Jul 30 10:39:20 camera-testing kernel:  __vb2_queue_cancel+0xe0/0x2d8 [videobuf2_common]
Jul 30 10:39:20 camera-testing kernel:  vb2_core_streamoff+0x2c/0xd0 [videobuf2_common]
Jul 30 10:39:20 camera-testing kernel:  vb2_ioctl_streamoff+0x54/0xa0 [videobuf2_v4l2]
Jul 30 10:39:20 camera-testing kernel:  v4l_streamoff+0x2c/0x40 [videodev]
Jul 30 10:39:20 camera-testing kernel:  __video_do_ioctl+0x170/0x3e0 [videodev]
Jul 30 10:39:20 camera-testing kernel:  video_usercopy+0x1c0/0x790 [videodev]
Jul 30 10:39:20 camera-testing kernel:  video_ioctl2+0x20/0x38 [videodev]
Jul 30 10:39:20 camera-testing kernel:  v4l2_ioctl+0x48/0x70 [videodev]
Jul 30 10:39:20 camera-testing kernel:  __arm64_sys_ioctl+0xb4/0x100
Jul 30 10:39:20 camera-testing kernel:  invoke_syscall+0x50/0x120
Jul 30 10:39:20 camera-testing kernel:  el0_svc_common.constprop.0+0x48/0xf0
Jul 30 10:39:20 camera-testing kernel:  do_el0_svc+0x24/0x38
Jul 30 10:39:20 camera-testing kernel:  el0_svc+0x30/0x100
Jul 30 10:39:20 camera-testing kernel:  el0t_64_sync_handler+0x100/0x130
Jul 30 10:39:20 camera-testing kernel:  el0t_64_sync+0x190/0x198
Jul 30 10:39:20 camera-testing kernel: Code: 9ad52400 36000900 f8647833 b40008d3 (f9402274) 
Jul 30 10:39:20 camera-testing kernel: ---[ end trace 0000000000000000 ]---

At this point the system has locked up, as we observed by seeing all logging stop at this point for our custom app that invokes rpicam-still. The only additional logging we have is kernel logs from journalctl after this point.

Error Log 2: __kmalloc_cache_noprof

Jul 30 10:41:10 camera-testing kernel: Unable to handle kernel paging request at virtual address 00f7c00000f78480
Jul 30 10:41:10 camera-testing kernel: Mem abort info:
Jul 30 10:41:10 camera-testing kernel:   ESR = 0x0000000096000004
Jul 30 10:41:10 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 30 10:41:11 camera-testing kernel:   SET = 0, FnV = 0
Jul 30 10:41:11 camera-testing kernel:   EA = 0, S1PTW = 0
Jul 30 10:41:11 camera-testing kernel:   FSC = 0x04: level 0 translation fault
Jul 30 10:41:11 camera-testing kernel: Data abort info:
Jul 30 10:41:11 camera-testing kernel:   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
Jul 30 10:41:11 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Jul 30 10:41:11 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 30 10:41:11 camera-testing kernel: [00f7c00000f78480] address between user and kernel address ranges
Jul 30 10:41:11 camera-testing kernel: Internal error: Oops: 0000000096000004 [#2] PREEMPT SMP
Jul 30 10:41:11 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc brcmfmac_wcc vc4 snd_soc_hdmi_codec ov64a40 bu64754 drm_display_helper v4l2_cci regmap_i2c brcmfmac hci_uart cec brcmutil drm_dma_helper btbcm bluetooth snd_soc_core cfg80211 aes_ce_blk aes_ce_cipher ghash_ce snd_compress gf128mul ecdh_generic rpi_hevc_dec snd_pcm_dmaengine pisp_be sha2_ce ecc rp1_cfe snd_pcm sha256_arm64 rfkill sha1_ce v4l2_mem2mem snd_timer libaes v4l2_fwnode v4l2_async videobuf2_dma_contig v3d snd videobuf2_memops sha1_generic w1_gpio videobuf2_v4l2 raspberrypi_hwmon gpu_sched wire videodev drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_fw raspberrypi_gpiomem rp1_adc rp1_mailbox nvmem_rmem uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6 spidev spi_bcm2835 i2c_brcmstb gpio_keys i2c_designware_platform i2c_designware_core
Jul 30 10:41:11 camera-testing kernel: CPU: 2 UID: 0 PID: 97727 Comm: (imedated) Tainted: G      D            6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Jul 30 10:41:11 camera-testing kernel: Tainted: [D]=DIE
Jul 30 10:41:11 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Jul 30 10:41:11 camera-testing kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Jul 30 10:41:11 camera-testing kernel: pc : __kmalloc_cache_noprof+0x88/0x2c8
Jul 30 10:41:11 camera-testing kernel: lr : do_seccomp+0x35c/0xae8
Jul 30 10:41:11 camera-testing kernel: sp : ffffc00081bd3d20
Jul 30 10:41:11 camera-testing kernel: x29: ffffc00081bd3d30 x28: ffff800003306300 x27: 0000000000000000
Jul 30 10:41:11 camera-testing kernel: x26: 0000000000000000 x25: ffffd06fcf6303e8 x24: ffffd06fce5d1994
Jul 30 10:41:11 camera-testing kernel: x23: ffffd06fce5d1994 x22: 00000000000000e0 x21: 0000000000002dc0
Jul 30 10:41:11 camera-testing kernel: x20: 00f7c00000f78400 x19: ffff800002004500 x18: 0000000000000000
Jul 30 10:41:11 camera-testing kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 000055562babc150
Jul 30 10:41:11 camera-testing kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
Jul 30 10:41:11 camera-testing kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fce5d1994
Jul 30 10:41:11 camera-testing kernel: x8 : 000055562bad1ff0 x7 : 000000000000001b x6 : ffffc00081bd3d98
Jul 30 10:41:11 camera-testing kernel: x5 : ffffc00081bd3d98 x4 : 0000000000000000 x3 : 000000000006d31a
Jul 30 10:41:11 camera-testing kernel: x2 : 0000000000000080 x1 : 00f7c00000f78400 x0 : ffff800002004500
Jul 30 10:41:11 camera-testing kernel: Call trace:
Jul 30 10:41:11 camera-testing kernel:  __kmalloc_cache_noprof+0x88/0x2c8
Jul 30 10:41:11 camera-testing kernel:  do_seccomp+0x35c/0xae8
Jul 30 10:41:11 camera-testing kernel:  __arm64_sys_seccomp+0x28/0x40
Jul 30 10:41:11 camera-testing kernel:  invoke_syscall+0x50/0x120
Jul 30 10:41:11 camera-testing kernel:  el0_svc_common.constprop.0+0xc8/0xf0
Jul 30 10:41:11 camera-testing kernel:  do_el0_svc+0x24/0x38
Jul 30 10:41:11 camera-testing kernel:  el0_svc+0x30/0x100
Jul 30 10:41:11 camera-testing kernel:  el0t_64_sync_handler+0x100/0x130
Jul 30 10:41:11 camera-testing kernel:  el0t_64_sync+0x190/0x198
Jul 30 10:41:11 camera-testing kernel: Code: f9002bf9 aa1403e1 aa1303e0 b9402a62 (f8626a99) 
Jul 30 10:41:11 camera-testing kernel: ---[ end trace 0000000000000000 ]---

Error Log 3: __kmalloc_noprof

Jul 30 10:42:53 camera-testing kernel: Unable to handle kernel paging request at virtual address 00f7c00000f78480
Jul 30 10:42:53 camera-testing kernel: Mem abort info:
Jul 30 10:42:53 camera-testing kernel:   ESR = 0x0000000096000004
Jul 30 10:42:53 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 30 10:42:54 camera-testing kernel:   SET = 0, FnV = 0
Jul 30 10:42:54 camera-testing kernel:   EA = 0, S1PTW = 0
Jul 30 10:42:54 camera-testing kernel:   FSC = 0x04: level 0 translation fault
Jul 30 10:42:54 camera-testing kernel: Data abort info:
Jul 30 10:42:54 camera-testing kernel:   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
Jul 30 10:42:54 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Jul 30 10:42:54 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 30 10:42:54 camera-testing kernel: [00f7c00000f78480] address between user and kernel address ranges
Jul 30 10:42:54 camera-testing kernel: Internal error: Oops: 0000000096000004 [#3] PREEMPT SMP
Jul 30 10:42:54 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc brcmfmac_wcc vc4 snd_soc_hdmi_codec ov64a40 bu64754 drm_display_helper v4l2_cci regmap_i2c brcmfmac hci_uart cec brcmutil drm_dma_helper btbcm bluetooth snd_soc_core cfg80211 aes_ce_blk aes_ce_cipher ghash_ce snd_compress gf128mul ecdh_generic rpi_hevc_dec snd_pcm_dmaengine pisp_be sha2_ce ecc rp1_cfe snd_pcm sha256_arm64 rfkill sha1_ce v4l2_mem2mem snd_timer libaes v4l2_fwnode v4l2_async videobuf2_dma_contig v3d snd videobuf2_memops sha1_generic w1_gpio videobuf2_v4l2 raspberrypi_hwmon gpu_sched wire videodev drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_fw raspberrypi_gpiomem rp1_adc rp1_mailbox nvmem_rmem uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6 spidev spi_bcm2835 i2c_brcmstb gpio_keys i2c_designware_platform i2c_designware_core
Jul 30 10:42:54 camera-testing kernel: CPU: 2 UID: 0 PID: 97874 Comm: (imedated) Tainted: G      D            6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Jul 30 10:42:54 camera-testing kernel: Tainted: [D]=DIE
Jul 30 10:42:54 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Jul 30 10:42:54 camera-testing kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Jul 30 10:42:54 camera-testing kernel: pc : __kmalloc_noprof+0xc8/0x3a8
Jul 30 10:42:54 camera-testing kernel: lr : bpf_convert_filter+0x70/0xb70
Jul 30 10:42:54 camera-testing kernel: sp : ffffc00082f33ad0
Jul 30 10:42:54 camera-testing kernel: x29: ffffc00082f33ae0 x28: ffff8000098dce00 x27: 0000000000000000
Jul 30 10:42:54 camera-testing kernel: x26: 0000000000000000 x25: ffffc00082450048 x24: 0000000000000076
Jul 30 10:42:54 camera-testing kernel: x23: 00f7c00000f78400 x22: ffffd06fcefa5750 x21: 0000000000002dc0
Jul 30 10:42:54 camera-testing kernel: x20: ffff800002004500 x19: 00000000000000ec x18: 0000000000000000
Jul 30 10:42:54 camera-testing kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000040025
Jul 30 10:42:54 camera-testing kernel: x14: 0000001400000021 x13: 0005006100000006 x12: 0000000101000025
Jul 30 10:42:54 camera-testing kernel: x11: 7fff000000000006 x10: 0005006100000006 x9 : ffffd06fcefa5750
Jul 30 10:42:54 camera-testing kernel: x8 : ffffc00082f33c68 x7 : 0000000000000000 x6 : 0000000000000000
Jul 30 10:42:54 camera-testing kernel: x5 : 000000000000003a x4 : ffffc00082f33cdf x3 : 000000000006d4f2
Jul 30 10:42:54 camera-testing kernel: x2 : 0000000000000080 x1 : 00f7c00000f78400 x0 : ffff800002004500
Jul 30 10:42:54 camera-testing kernel: Call trace:
Jul 30 10:42:54 camera-testing kernel:  __kmalloc_noprof+0xc8/0x3a8
Jul 30 10:42:54 camera-testing kernel:  bpf_convert_filter+0x70/0xb70
Jul 30 10:42:54 camera-testing kernel:  bpf_prepare_filter+0x460/0x598
Jul 30 10:42:54 camera-testing kernel:  bpf_prog_create_from_user+0xdc/0x1c0
Jul 30 10:42:54 camera-testing kernel:  do_seccomp+0x39c/0xae8
Jul 30 10:42:54 camera-testing kernel:  __arm64_sys_seccomp+0x28/0x40
Jul 30 10:42:54 camera-testing kernel:  invoke_syscall+0x50/0x120
Jul 30 10:42:54 camera-testing kernel:  el0_svc_common.constprop.0+0xc8/0xf0
Jul 30 10:42:54 camera-testing kernel:  do_el0_svc+0x24/0x38
Jul 30 10:42:54 camera-testing kernel:  el0_svc+0x30/0x100
Jul 30 10:42:54 camera-testing kernel:  el0t_64_sync_handler+0x100/0x130
Jul 30 10:42:54 camera-testing kernel:  el0t_64_sync+0x190/0x198
Jul 30 10:42:54 camera-testing kernel: Code: 540013a0 b9402a82 aa1703e1 aa1403e0 (f8626af8) 
Jul 30 10:42:54 camera-testing kernel: ---[ end trace 0000000000000000 ]---

Error Log 4: __kmalloc_cache_noprof (again)

Jul 30 10:43:20 camera-testing kernel: Unable to handle kernel paging request at virtual address 00f7c00000f78480
Jul 30 10:43:20 camera-testing kernel: Mem abort info:
Jul 30 10:43:20 camera-testing kernel:   ESR = 0x0000000096000004
Jul 30 10:43:20 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Jul 30 10:43:20 camera-testing kernel:   SET = 0, FnV = 0
Jul 30 10:43:20 camera-testing kernel:   EA = 0, S1PTW = 0
Jul 30 10:43:20 camera-testing kernel:   FSC = 0x04: level 0 translation fault
Jul 30 10:43:20 camera-testing kernel: Data abort info:
Jul 30 10:43:20 camera-testing kernel:   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
Jul 30 10:43:20 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Jul 30 10:43:20 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Jul 30 10:43:20 camera-testing kernel: [00f7c00000f78480] address between user and kernel address ranges
Jul 30 10:43:20 camera-testing kernel: Internal error: Oops: 0000000096000004 [#4] PREEMPT SMP
Jul 30 10:43:20 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc brcmfmac_wcc vc4 snd_soc_hdmi_codec ov64a40 bu64754 drm_display_helper v4l2_cci regmap_i2c brcmfmac hci_uart cec brcmutil drm_dma_helper btbcm bluetooth snd_soc_core cfg80211 aes_ce_blk aes_ce_cipher ghash_ce snd_compress gf128mul ecdh_generic rpi_hevc_dec snd_pcm_dmaengine pisp_be sha2_ce ecc rp1_cfe snd_pcm sha256_arm64 rfkill sha1_ce v4l2_mem2mem snd_timer libaes v4l2_fwnode v4l2_async videobuf2_dma_contig v3d snd videobuf2_memops sha1_generic w1_gpio videobuf2_v4l2 raspberrypi_hwmon gpu_sched wire videodev drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_fw raspberrypi_gpiomem rp1_adc rp1_mailbox nvmem_rmem uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks dm_mod backlight ip_tables x_tables ipv6 spidev spi_bcm2835 i2c_brcmstb gpio_keys i2c_designware_platform i2c_designware_core
Jul 30 10:43:20 camera-testing kernel: CPU: 2 UID: 0 PID: 512 Comm: w1_bus_master1 Tainted: G      D            6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Jul 30 10:43:20 camera-testing kernel: Tainted: [D]=DIE
Jul 30 10:43:20 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Jul 30 10:43:20 camera-testing kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Jul 30 10:43:20 camera-testing kernel: pc : __kmalloc_cache_noprof+0x88/0x2c8
Jul 30 10:43:20 camera-testing kernel: lr : device_add+0x40c/0x758
Jul 30 10:43:20 camera-testing kernel: sp : ffffc00080ec3b80
Jul 30 10:43:20 camera-testing kernel: x29: ffffc00080ec3b90 x28: 00000000000000f0 x27: ffffd06fbd709230
Jul 30 10:43:20 camera-testing kernel: x26: 0000000000000001 x25: ffff800008902008 x24: ffffd06fcecd63ec
Jul 30 10:43:20 camera-testing kernel: x23: ffffd06fcecd63ec x22: 00000000000000d8 x21: 0000000000000dc0
Jul 30 10:43:20 camera-testing kernel: x20: 00f7c00000f78400 x19: ffff800002004500 x18: 000000000000000c
Jul 30 10:43:20 camera-testing kernel: x17: 35302e3030303030 x16: ffffd06fcecd6738 x15: ffffc00080ec3af0
Jul 30 10:43:20 camera-testing kernel: x14: ffff800008902028 x13: ffff80000890200b x12: 67657220746f6e20
Jul 30 10:43:20 camera-testing kernel: x11: 73692035302e3030 x10: 000000000000002d x9 : ffffd06fcecd63ec
Jul 30 10:43:20 camera-testing kernel: x8 : ffff800008902198 x7 : 0000000000000000 x6 : 000fffffffe00000
Jul 30 10:43:20 camera-testing kernel: x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000006d4f2
Jul 30 10:43:20 camera-testing kernel: x2 : 0000000000000080 x1 : 00f7c00000f78400 x0 : ffff800002004500
Jul 30 10:43:20 camera-testing kernel: Call trace:
Jul 30 10:43:20 camera-testing kernel:  __kmalloc_cache_noprof+0x88/0x2c8
Jul 30 10:43:20 camera-testing kernel:  device_add+0x40c/0x758
Jul 30 10:43:20 camera-testing kernel:  device_register+0x28/0x40
Jul 30 10:43:20 camera-testing kernel:  w1_attach_slave_device+0x1bc/0x458 [wire]
Jul 30 10:43:20 camera-testing kernel:  w1_slave_found+0x17c/0x1e8 [wire]
Jul 30 10:43:20 camera-testing kernel:  w1_search+0x1e0/0x250 [wire]
Jul 30 10:43:20 camera-testing kernel:  w1_search_devices+0x54/0x60 [wire]
Jul 30 10:43:20 camera-testing kernel:  w1_search_process_cb+0x84/0x170 [wire]
Jul 30 10:43:20 camera-testing kernel:  w1_process+0x1b0/0x220 [wire]
Jul 30 10:43:20 camera-testing kernel:  kthread+0x11c/0x128
Jul 30 10:43:20 camera-testing kernel:  ret_from_fork+0x10/0x20
Jul 30 10:43:20 camera-testing kernel: Code: f9002bf9 aa1403e1 aa1303e0 b9402a62 (f8626a99) 
Jul 30 10:43:20 camera-testing kernel: ---[ end trace 0000000000000000 ]---
Jul 30 10:46:18 camera-testing kernel: Unable to handle kernel paging request at virtual address 00f7c00000f78480

The above log is the last line of logging we have in the system before it went completely unresponsive.

Additional context
At the time of Error Log 1 at 10:39:20, this is the last moment at which our app interacted with the system. Our app's logging ends at 10:39:20, and so we only have these kernel logs after the fact to go off of. Our app shows a Called Process Error with SIGABRT, and no more logging after this.

Called Process Error: Command '['/usr/bin/libcamera-still', '--width', '4624', '--height', '3472', '--camera', '0', '-o', '/dev/shm/tmp_0.jpg', '-n', '-t', '3000', '--no-raw', '--encoding', 'jpg', '--quality', '100', '--rotation', '0']' died with <Signals.SIGABRT: 6>.

• The rpi's uname is 6.12.34+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux.
• We increased the CMA buffer to 512MiB (this used to be the default, but for some reason has changed), but that did not help.
• We increased the page file to 8GB and that also did not help.

@wallarug and I have tried troubleshooting this problem over the past 2-3 weeks but to no avail. This problem has only appeared since we updated to the Linux 2025-06-26 rpi firmware. Any help with this issue would be immensely appreciated, we are happy to provide any further information and welcome any tips for investigation.

[naushir]

Note that you should be using rpicam-still in place of libcamera-still. What does the terminal output of rpicam-still show when the crash occurs?

Can you also share the output of rpicam-still --version please?

[naushir]

We increased the CMA buffer to 512MiB (this used to be the default, but for some reason has changed), but that did not help.

Raspberry Pi 5 does not use CMA allocations for camera framebuffers - you should revert this back to default.

[goombado]

Thanks for getting back to me!

rpicam-apps version:

rpicam-apps build: v1.7.0 5a3f5965aca9 30-04-2025 (11:42:35)
rpicam-apps capabilites: egl:1 qt:1 drm:1 libav:1
libcamera build: v0.5.0+59-d83ff0a4

Happy to switch to rpicam-still, although I can see on my machine that libcamera-still is directly symlinked to rpicam-still, so hopefully that isn't a factor in this issue.

We can't see any terminal output for rpicam-still unfortunately, as we get an entire system/kernel crash. The closest I can get to terminal output and logging is from our app that invokes rpicam-still, showing a SIGABRT.

Called Process Error: Command '['/usr/bin/libcamera-still', '--width', '4624', '--height', '3472', '--camera', '0', '-o', '/dev/shm/tmp_0.jpg', '-n', '-t', '3000', '--no-raw', '--encoding', 'jpg', '--quality', '100', '--rotation', '0']' died with <Signals.SIGABRT: 6>.

It seems like the only thing we have to go off of is the kernel logs. rpicam-still will work exactly as intended for ~10 hours before inevitably seeing these kernel errors.

Will will also revert the CMA back to default.

Thank you again for looking into this!

[naushir]

May I suggest you update rpicam-apps and libcamera to the latest version please? We have recently fixed a bug (more information at #799) that would cause random lockups, similar to what you might be seeing here.

[goombado]

Sorry, assumed we were using the latest version. Will do and will report back, thanks!

[wallarug]

May I suggest you update rpicam-apps and libcamera to the latest version please? We have recently fixed a bug (more information at #799) that would cause random lockups, similar to what you might be seeing here.

I just updated one of our production cameras. We will see how it goes on the new rpiapps. I am hopeful after you pointed us in the direction of that Issue/PR chain that this was a similar or same issue.

We will report back in 24 - 48 hours if we are still seeing issues (it normally takes that long for them to appear).

Current Version:

rpicam-apps build: v1.8.1 eb293b055248 21-07-2025 (11:50:44)
rpicam-apps capabilites: egl:0 qt:0 drm:1 libav:0
libcamera build: v0.5.1+100-e53bdf1f

[wallarug]

This recommendation did not fix our issue. It is still occurring. Will report back in with new logs on Monday.

[wallarug]

From what I can see in our logging for the software, it appears the camera rpicam-still could have caused the lock up of the system. It doesn't make sense to me though that rpicam-still under certain conditions causes this to happen. More to the point - it seems to happen after ~3 days (consistently). I was checking the camera periodically, there did not appear to be any 'zombie' or 'dead' processes when I checked it, unlike before.

Interestingly, in all our logs when this happens - it looks like all processes crash (even non-celery ones) and attempt to soft-restart. They do restart but then very quickly the system stops working.

This one has puzzled us for about a month now. We even re-wrote our celery stack to be more 'restrictive' to stop a flood of tasks being added. That did not solve the problem. So now we think it must be something further down the system stack (such as rpicam-still).

See the logs below (at least what we could gather before the system locked up)...

celery.log (Timezone: +8 GMT)

Aug 02 16:14:25 xx sh[1367]: [2025-08-02 16:14:25,210: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
Aug 02 16:14:25 xx kernel: rp1-cfe 1f00128000.csi: Using a link rate of 720 Mbps
Aug 02 16:14:25 xx kernel: rp1-cfe 1f00110000.csi: Using a link rate of 720 Mbps
Aug 02 16:14:30 xx sh[1367]: [2025-08-02 16:14:30,211: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
Aug 02 16:14:30 xx kernel: rp1-cfe 1f00128000.csi: Using a link rate of 720 Mbps
Aug 02 16:14:30 xx kernel: rp1-cfe 1f00110000.csi: Using a link rate of 720 Mbps
EOF

beat.log (Timezone: +8 GMT)

[2025-08-02 16:14:22,466: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task Get CPU Temperature Reading (environment.tasks.record_temp)
[2025-08-02 16:14:25,210: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
[2025-08-02 16:14:30,211: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
[2025-08-02 16:14:35,211: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
[2025-08-02 16:14:37,467: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task Get CPU Temperature Reading (environment.tasks.record_temp)
[2025-08-02 16:14:40,211: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
[2025-08-02 16:14:34,089: INFO/MainProcess] celery.beat.start[beat] beat: Starting...
[2025-08-02 16:14:35,281: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2025-08-02 16:14:35,289: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task Get CPU Temperature Reading (environment.tasks.record_temp)
[2025-08-02 16:14:35,483: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task upload-pending-images (management.tasks.upload_pending_images)
[2025-08-02 16:14:35,486: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)
[2025-08-02 16:14:35,492: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task internet-health-check (management.tasks.check_internet_health_task)
[2025-08-02 16:14:40,487: INFO/MainProcess] celery.beat.apply_entry[beat] Scheduler: Sending due task camera-takes-images (tasks.timelapse_runtime)

Database Logging System (not celery driven)

[2025-08-02 16:13:09,627: INFO] Database has 11 active connections
[2025-08-02 16:14:09,926: INFO] Database has 10 active connections
[2025-08-02 16:14:23,710: INFO] Starting database connection monitor
[2025-08-02 16:14:24,521: INFO] Database has 6 active connections
[2025-08-03 09:20:53,377: INFO] Database has 8 active connections
[2025-08-03 09:21:53,782: INFO] Database has 7 active connections
[2025-08-03 09:22:54,176: INFO] Database has 7 active connections
[2025-08-03 09:23:54,477: INFO] Database has 7 active connections

Django Server log (not celery driven)

[2025-07-30 23:14:56,024: WARNING/MainProcess] django.request.log_response[log] Not Found: /favicon.ico
[2025-08-02 16:14:35,633: INFO/MainProcess] django.utils.autoreload.run_with_reloader[autoreload] Watching for file changes with StatReloader

I have slightly more verbose logging from celery, I am happy to share that via email / private message.

This one has stumped us. It seems it could be something camera software related, but we are at a dead end as to why all the system processes "soft restart" because of it.

Interested in your thoughts and opinions here,

[naushir]

To help debug this issue, let's try and reproduce it with minimal software overhead. Can you run the rpicam-still command on it's own in a loop, i.e. without any additional software running on top. And if possible, ideally on a fresh sdcard image with all the updates applied.

[wallarug]

To help debug this issue, let's try and reproduce it with minimal software overhead. Can you run the rpicam-still command on it's own in a loop, i.e. without any additional software running on top. And if possible, ideally on a fresh sdcard image with all the updates applied.

I can try this later today (it's 1.15 AM right now so I'm about to knock off).

Just FYI - we are still getting the orphaned processes that were reported in #799

[goombado]

Hi all,

Further looking into this issue after updating to the latest version of rpicam-apps, we have received another kernel panic directly related to rpicam-still:

Aug 04 17:42:27 rpi kernel: Unable to handle kernel paging request at virtual address 0000003f7fc00a08
Aug 04 17:42:27 rpi kernel: Mem abort info:
Aug 04 17:42:28 rpi kernel:   ESR = 0x0000000096000005
Aug 04 17:42:28 rpi kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Aug 04 17:42:28 rpi kernel:   SET = 0, FnV = 0
Aug 04 17:42:28 rpi kernel:   EA = 0, S1PTW = 0
Aug 04 17:42:28 rpi kernel:   FSC = 0x05: level 1 translation fault
Aug 04 17:42:28 rpi kernel: Data abort info:
Aug 04 17:42:28 rpi kernel:   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
Aug 04 17:42:28 rpi kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Aug 04 17:42:28 rpi kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Aug 04 17:42:28 rpi kernel: user pgtable: 16k pages, 47-bit VAs, pgdp=0000000100d10000
Aug 04 17:42:28 rpi kernel: [0000003f7fc00a08] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
Aug 04 17:42:28 rpi kernel: Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
Aug 04 17:42:28 rpi kernel: Modules linked in: nf_tables nfnetlink wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha binfmt_misc brcmfmac_wcc vc4 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper brcmfmac snd_soc_core ov64a40 brcmutil bu64754 cfg80211 v4l2_cci regmap_i2c snd_compress hci_uart aes_ce_blk snd_pcm_dmaengine btbcm aes_ce_cipher bluetooth snd_pcm ghash_ce rpi_hevc_dec gf128mul rp1_cfe pisp_be ecdh_generic sha2_ce snd_timer ecc sha256_arm64 snd rfkill v4l2_fwnode v4l2_mem2mem sha1_ce libaes v4l2_async videobuf2_dma_contig videobuf2_memops v3d videobuf2_v4l2 sha1_generic videodev gpu_sched raspberrypi_hwmon videobuf2_common drm_shmem_helper mc drm_kms_helper pwm_fan rp1_pio rp1_fw raspberrypi_gpiomem rp1_mailbox rp1_adc nvmem_rmem uio_pdrv_genirq uio drm drm_panel_orientation_quirks fuse backlight dm_mod ip_tables x_tables ipv6 spidev i2c_brcmstb spi_bcm2835 gpio_keys i2c_designware_platform i2c_designware_core
Aug 04 17:42:28 rpi kernel: CPU: 1 UID: 1000 PID: 177037 Comm: rpicam-still Not tainted 6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Aug 04 17:42:28 rpi kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Aug 04 17:42:28 rpi kernel: pstate: a0400009 (NzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 04 17:42:28 rpi kernel: pc : kfree+0x60/0x2d0
Aug 04 17:42:28 rpi kernel: lr : pispbe_prepare_job+0x2e8/0x540 [pisp_be]
Aug 04 17:42:28 rpi kernel: sp : ffffc0008414b9c0
Aug 04 17:42:28 rpi kernel: x29: ffffc0008414b9d0 x28: ffffd06fa7950938 x27: ffffd06fa7855078
Aug 04 17:42:28 rpi kernel: x26: 0000000000000000 x25: ffff8000c0f44000 x24: ffff8000c0f44200
Aug 04 17:42:28 rpi kernel: x23: ffff800006440098 x22: 00000000000000f7 x21: ffffd06fa79195b8
Aug 04 17:42:28 rpi kernel: x20: ffffffbfc00a0100 x19: 0000003f7fc00a00 x18: 0000000000000000
Aug 04 17:42:28 rpi kernel: x17: 0000000000000000 x16: ffffd06fce7626a0 x15: 00007fff5b6cc7c8
Aug 04 17:42:28 rpi kernel: x14: 0000000400000000 x13: 0000000000000000 x12: 0000000000000000
Aug 04 17:42:28 rpi kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fa79195b8
Aug 04 17:42:28 rpi kernel: x8 : ffffffbfc00a0200 x7 : 0000000000000000 x6 : 000000000000003f
Aug 04 17:42:28 rpi kernel: x5 : 0000000000000040 x4 : ffff800003d22690 x3 : 0000000000000004
Aug 04 17:42:28 rpi kernel: x2 : 0000000000000000 x1 : ffff800006443cf8 x0 : ffffffbfc0000000
Aug 04 17:42:28 rpi kernel: Call trace:
Aug 04 17:42:28 rpi kernel:  kfree+0x60/0x2d0
Aug 04 17:42:28 rpi kernel:  pispbe_prepare_job+0x2e8/0x540 [pisp_be]
Aug 04 17:42:28 rpi kernel:  pispbe_node_buffer_queue+0x50/0x90 [pisp_be]
Aug 04 17:42:28 rpi kernel:  __enqueue_in_driver+0x54/0xe8 [videobuf2_common]
Aug 04 17:42:28 rpi kernel:  vb2_core_qbuf+0x3c8/0x5c8 [videobuf2_common]
Aug 04 17:42:28 rpi kernel:  vb2_qbuf+0xb4/0x198 [videobuf2_v4l2]
Aug 04 17:42:28 rpi kernel:  vb2_ioctl_qbuf+0x68/0x1660 [videobuf2_v4l2]
Aug 04 17:42:28 rpi kernel:  v4l_qbuf+0x50/0x70 [videodev]
Aug 04 17:42:28 rpi kernel:  __video_do_ioctl+0x170/0x3e0 [videodev]
Aug 04 17:42:28 rpi kernel:  video_usercopy+0x1c0/0x790 [videodev]
Aug 04 17:42:28 rpi kernel:  video_ioctl2+0x20/0x38 [videodev]
Aug 04 17:42:28 rpi kernel:  v4l2_ioctl+0x48/0x70 [videodev]
Aug 04 17:42:28 rpi kernel:  __arm64_sys_ioctl+0xb4/0x100
Aug 04 17:42:28 rpi kernel:  invoke_syscall+0x50/0x120
Aug 04 17:42:28 rpi kernel:  el0_svc_common.constprop.0+0x48/0xf0
Aug 04 17:42:28 rpi kernel:  do_el0_svc+0x24/0x38
Aug 04 17:42:28 rpi kernel:  el0_svc+0x30/0x100
Aug 04 17:42:28 rpi kernel:  el0t_64_sync_handler+0x100/0x130
Aug 04 17:42:28 rpi kernel:  el0t_64_sync+0x190/0x198
Aug 04 17:42:28 rpi kernel: Code: b26287e0 d34efe73 f2dff7e0 8b131813 (f9400660) 
Aug 04 17:42:28 rpi kernel: ---[ end trace 0000000000000000 ]---

We are currently running the following script to simulate a full day of capture (~1000 images). It will take about 2.5 hours to run, so we will report back once it's done with any issues if they've arisen.

while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting camera capture 0" >> /var/log/camera_capture.log
    /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0 >> /var/log/camera_capture_cam0.log 2>&1 &
    echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting camera capture 1" >> /var/log/camera_capture.log
    /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0  >> /var/log/camera_capture_cam1.log 2>&1 &

    wait
    echo "$(date '+%Y-%m-%d %H:%M:%S') - Camera captures completed" >> /var/log/camera_capture.log

done

I will note as well that we are constantly monitoring the voltage level of the RPi using the internal voltage monitor, and we haven't been able identify this as a potential cause for the issues.

[goombado]

We didn't have to wait 2.5 hours, after 15 minutes of running rpicam-still on a loop the issue was reproduced

Aug 05 15:19:30 camera-testing kernel: Unable to handle kernel paging request at virtual address 0000415000000080
Aug 05 15:19:30 camera-testing kernel: Mem abort info:
Aug 05 15:19:30 camera-testing kernel:   ESR = 0x0000000096000005
Aug 05 15:19:30 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Aug 05 15:19:30 camera-testing kernel: rp1-cfe 1f00110000.csi: Using a link rate of 720 Mbps
Aug 05 15:19:30 camera-testing kernel:   SET = 0, FnV = 0
Aug 05 15:19:30 camera-testing kernel:   EA = 0, S1PTW = 0
Aug 05 15:19:30 camera-testing kernel:   FSC = 0x05: level 1 translation fault
Aug 05 15:19:30 camera-testing kernel: Data abort info:
Aug 05 15:19:30 camera-testing kernel:   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
Aug 05 15:19:30 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Aug 05 15:19:30 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Aug 05 15:19:30 camera-testing kernel: user pgtable: 16k pages, 47-bit VAs, pgdp=000000005138c000
Aug 05 15:19:30 camera-testing kernel: [0000415000000080] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
Aug 05 15:19:31 camera-testing kernel: Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
Aug 05 15:19:31 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc vc4 ov64a40 bu64754 brcmfmac_wcc v4l2_cci regmap_i2c brcmfmac snd_soc_hdmi_codec hci_uart btbcm drm_display_helper aes_ce_blk brcmutil cec bluetooth aes_ce_cipher cfg80211 drm_dma_helper ghash_ce snd_soc_core gf128mul sha2_ce snd_compress ecdh_generic rpi_hevc_dec sha256_arm64 ecc rp1_cfe pisp_be snd_pcm_dmaengine sha1_ce snd_pcm rfkill libaes v4l2_mem2mem v4l2_fwnode snd_timer v3d sha1_generic v4l2_async videobuf2_dma_contig snd videobuf2_memops raspberrypi_hwmon w1_gpio videobuf2_v4l2 gpu_sched videodev wire drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_adc rp1_mailbox rp1_fw raspberrypi_gpiomem nvmem_rmem uio_pdrv_genirq uio i2c_dev drm drm_panel_orientation_quirks dm_mod fuse backlight ip_tables x_tables ipv6 spidev i2c_brcmstb spi_bcm2835 gpio_keys i2c_designware_platform i2c_designware_core
Aug 05 15:19:31 camera-testing kernel: CPU: 1 UID: 0 PID: 182361 Comm: rpicam-still Not tainted 6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Aug 05 15:19:31 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Aug 05 15:19:31 camera-testing kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 05 15:19:31 camera-testing kernel: pc : __kmalloc_noprof+0xc8/0x3a8
Aug 05 15:19:31 camera-testing kernel: lr : vb2_core_allocated_buffers_storage+0x78/0x90 [videobuf2_common]
Aug 05 15:19:31 camera-testing kernel: sp : ffffc00080973a60
Aug 05 15:19:31 camera-testing kernel: x29: ffffc00080973a70 x28: ffffd06fc6064938 x27: ffffd06fc5e78f98
Aug 05 15:19:31 camera-testing kernel: x26: 0000000000000000 x25: 0000000000000000 x24: ffff800003825a00
Aug 05 15:19:31 camera-testing kernel: x23: 0000415000000000 x22: ffffd06fc5d19e80 x21: 0000000000000dc0
Aug 05 15:19:31 camera-testing kernel: x20: ffff800002004500 x19: 0000000000000100 x18: 0000000000000000
Aug 05 15:19:31 camera-testing kernel: x17: 0000000000000000 x16: ffffd06fce7667d8 x15: 00007fff3cf98c50
Aug 05 15:19:31 camera-testing kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
Aug 05 15:19:31 camera-testing kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fc5d19e80
Aug 05 15:19:31 camera-testing kernel: x8 : ffff800003825a40 x7 : 0000000000000000 x6 : 0000000000000010
Aug 05 15:19:31 camera-testing kernel: x5 : 0000000000000020 x4 : 0000000000000000 x3 : 000000000002f295
Aug 05 15:19:31 camera-testing kernel: x2 : 0000000000000080 x1 : 0000415000000000 x0 : ffff800002004500
Aug 05 15:19:31 camera-testing kernel: Call trace:
Aug 05 15:19:31 camera-testing kernel:  __kmalloc_noprof+0xc8/0x3a8
Aug 05 15:19:31 camera-testing kernel:  vb2_core_allocated_buffers_storage+0x78/0x90 [videobuf2_common]
Aug 05 15:19:31 camera-testing kernel:  vb2_core_reqbufs+0x10c/0x4b8 [videobuf2_common]
Aug 05 15:19:31 camera-testing kernel:  vb2_ioctl_reqbufs+0xdc/0x110 [videobuf2_v4l2]
Aug 05 15:19:31 camera-testing kernel:  v4l_reqbufs+0x7c/0xa0 [videodev]
Aug 05 15:19:31 camera-testing kernel:  __video_do_ioctl+0x170/0x3e0 [videodev]
Aug 05 15:19:31 camera-testing kernel:  video_usercopy+0x1c0/0x790 [videodev]
Aug 05 15:19:31 camera-testing kernel:  video_ioctl2+0x20/0x38 [videodev]
Aug 05 15:19:31 camera-testing kernel:  v4l2_ioctl+0x48/0x70 [videodev]
Aug 05 15:19:31 camera-testing kernel:  __arm64_sys_ioctl+0xb4/0x100
Aug 05 15:19:31 camera-testing kernel:  invoke_syscall+0x50/0x120
Aug 05 15:19:31 camera-testing kernel:  el0_svc_common.constprop.0+0x48/0xf0
Aug 05 15:19:31 camera-testing kernel:  do_el0_svc+0x24/0x38
Aug 05 15:19:31 camera-testing kernel:  el0_svc+0x30/0x100
Aug 05 15:19:31 camera-testing kernel:  el0t_64_sync_handler+0x100/0x130
Aug 05 15:19:31 camera-testing kernel:  el0t_64_sync+0x190/0x198
Aug 05 15:19:31 camera-testing kernel: Code: 540013a0 b9402a82 aa1703e1 aa1403e0 (f8626af8)
Aug 05 15:19:31 camera-testing kernel: ---[ end trace 0000000000000000 ]---

It appears this also happened with the second rpicam-still process, but instead the call trace refers to __kmalloc_cache_noprof

Aug 05 15:19:32 camera-testing kernel: Unable to handle kernel paging request at virtual address 0000415000000080
Aug 05 15:19:32 camera-testing kernel: Mem abort info:
Aug 05 15:19:32 camera-testing kernel:   ESR = 0x0000000096000005
Aug 05 15:19:32 camera-testing kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Aug 05 15:19:32 camera-testing kernel:   SET = 0, FnV = 0
Aug 05 15:19:32 camera-testing kernel:   EA = 0, S1PTW = 0
Aug 05 15:19:32 camera-testing kernel:   FSC = 0x05: level 1 translation fault
Aug 05 15:19:32 camera-testing kernel: Data abort info:
Aug 05 15:19:32 camera-testing kernel:   ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
Aug 05 15:19:32 camera-testing kernel:   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
Aug 05 15:19:32 camera-testing kernel:   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
Aug 05 15:19:32 camera-testing kernel: user pgtable: 16k pages, 47-bit VAs, pgdp=0000000042060000
Aug 05 15:19:32 camera-testing kernel: [0000415000000080] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
Aug 05 15:19:32 camera-testing kernel: Internal error: Oops: 0000000096000005 [#2] PREEMPT SMP
Aug 05 15:19:32 camera-testing kernel: Modules linked in: wireguard libchacha20poly1305 chacha_neon poly1305_neon ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha 8021q garp stp llc binfmt_misc vc4 ov64a40 bu64754 brcmfmac_wcc v4l2_cci regmap_i2c brcmfmac snd_soc_hdmi_codec hci_uart btbcm drm_display_helper aes_ce_blk brcmutil cec bluetooth aes_ce_cipher cfg80211 drm_dma_helper ghash_ce snd_soc_core gf128mul sha2_ce snd_compress ecdh_generic rpi_hevc_dec sha256_arm64 ecc rp1_cfe pisp_be snd_pcm_dmaengine sha1_ce snd_pcm rfkill libaes v4l2_mem2mem v4l2_fwnode snd_timer v3d sha1_generic v4l2_async videobuf2_dma_contig snd videobuf2_memops raspberrypi_hwmon w1_gpio videobuf2_v4l2 gpu_sched videodev wire drm_shmem_helper cn drm_kms_helper videobuf2_common pwm_fan mc rp1_pio rp1_adc rp1_mailbox rp1_fw raspberrypi_gpiomem nvmem_rmem uio_pdrv_genirq uio i2c_dev drm drm_panel_orientation_quirks dm_mod fuse backlight ip_tables x_tables ipv6 spidev i2c_brcmstb spi_bcm2835 gpio_keys i2c_designware_platform i2c_designware_core
Aug 05 15:19:32 camera-testing kernel: CPU: 1 UID: 0 PID: 182360 Comm: rpicam-still Tainted: G      D            6.12.34+rpt-rpi-2712 #1  Debian 1:6.12.34-1+rpt1~bookworm
Aug 05 15:19:32 camera-testing kernel: Tainted: [D]=DIE
Aug 05 15:19:32 camera-testing kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
Aug 05 15:19:32 camera-testing kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 05 15:19:32 camera-testing kernel: pc : __kmalloc_cache_noprof+0x88/0x2c8
Aug 05 15:19:32 camera-testing kernel: lr : pispbe_prepare_job+0x90/0x540 [pisp_be]
Aug 05 15:19:32 camera-testing kernel: sp : ffffc00080853a30
Aug 05 15:19:32 camera-testing kernel: x29: ffffc00080853a40 x28: ffffd06fc6064938 x27: ffffd06fc5e790d8
Aug 05 15:19:32 camera-testing kernel: x26: 0000000000000000 x25: ffff800003823cd8 x24: ffffd06fc603d360
Aug 05 15:19:32 camera-testing kernel: x23: ffffd06fc603d360 x22: 00000000000000d8 x21: 0000000000000dc0
Aug 05 15:19:32 camera-testing kernel: x20: 0000415000000000 x19: ffff800002004500 x18: 0000000000000000
Aug 05 15:19:32 camera-testing kernel: x17: 0000000000000000 x16: ffffd06fce765a00 x15: 00007fff34047240
Aug 05 15:19:32 camera-testing kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
Aug 05 15:19:32 camera-testing kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fc603d360
Aug 05 15:19:32 camera-testing kernel: x8 : ffffc00080853ad8 x7 : 0000000000000000 x6 : 0000000000000000
Aug 05 15:19:32 camera-testing kernel: x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000002f295
Aug 05 15:19:32 camera-testing kernel: x2 : 0000000000000080 x1 : 0000415000000000 x0 : ffff800002004500
Aug 05 15:19:32 camera-testing kernel: Call trace:
Aug 05 15:19:32 camera-testing kernel:  __kmalloc_cache_noprof+0x88/0x2c8
Aug 05 15:19:32 camera-testing kernel:  pispbe_prepare_job+0x90/0x540 [pisp_be]
Aug 05 15:19:32 camera-testing kernel:  pispbe_node_start_streaming+0x8c/0x1a0 [pisp_be]
Aug 05 15:19:32 camera-testing kernel:  vb2_start_streaming+0x74/0x190 [videobuf2_common]
Aug 05 15:19:32 camera-testing kernel:  vb2_core_streamon+0x108/0x1d8 [videobuf2_common]
Aug 05 15:19:32 camera-testing kernel:  vb2_ioctl_streamon+0x54/0xa0 [videobuf2_v4l2]
Aug 05 15:19:32 camera-testing kernel:  v4l_streamon+0x2c/0x40 [videodev]
Aug 05 15:19:32 camera-testing kernel:  __video_do_ioctl+0x170/0x3e0 [videodev]
Aug 05 15:19:32 camera-testing kernel:  video_usercopy+0x1c0/0x790 [videodev]
Aug 05 15:19:32 camera-testing kernel:  video_ioctl2+0x20/0x38 [videodev]
Aug 05 15:19:32 camera-testing kernel:  v4l2_ioctl+0x48/0x70 [videodev]
Aug 05 15:19:32 camera-testing kernel:  __arm64_sys_ioctl+0xb4/0x100
Aug 05 15:19:32 camera-testing kernel:  invoke_syscall+0x50/0x120
Aug 05 15:19:32 camera-testing kernel:  el0_svc_common.constprop.0+0x48/0xf0
Aug 05 15:19:32 camera-testing kernel:  do_el0_svc+0x24/0x38
Aug 05 15:19:32 camera-testing kernel:  el0_svc+0x30/0x100
Aug 05 15:19:32 camera-testing kernel:  el0t_64_sync_handler+0x100/0x130
Aug 05 15:19:32 camera-testing kernel:  el0t_64_sync+0x190/0x198
Aug 05 15:19:32 camera-testing kernel: Code: f9002bf9 aa1403e1 aa1303e0 b9402a62 (f8626a99)
Aug 05 15:19:32 camera-testing kernel: ---[ end trace 0000000000000000 ]---

We also captured stdout and stderr of the rpicam-still processes. This is the last thing we got, and it all looks pretty normal. We had been seeing the Mising parameter lines since the script began running, so this was "expected" behaviour.

Still capture image received
[2:44:00.308629437] [182351]  INFO Camera camera_manager.cpp:326 libcamera v0.5.1+100-e53bdf1f
[2:44:00.320242058] [182360]  INFO RPI pisp.cpp:720 libpisp version v1.2.1 981977ff21f3 29-04-2025 (14:13:50)
[2:44:00.330645808] [182360]  WARN RPiAf af.cpp:77 Missing parameter "retrigger_ratio"
[2:44:00.330680678] [182360]  WARN RPiAf af.cpp:77 Missing parameter "retrigger_delay"
[2:44:00.330687400] [182360]  WARN RPiAf af.cpp:77 Missing parameter "check_for_ir"
[2:44:00.331599142] [182360]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@1000120000/rp1/i2c@88000/ov64a40@36 to CFE device /dev/media0 and ISP device /dev/media2 using PiSP variant BCM2712_C0
[2:44:00.331664809] [182360]  INFO RPI pisp.cpp:720 libpisp version v1.2.1 981977ff21f3 29-04-2025 (14:13:50)
[2:44:00.340127519] [182360]  WARN RPiAf af.cpp:77 Missing parameter "retrigger_ratio"
[2:44:00.340147797] [182360]  WARN RPiAf af.cpp:77 Missing parameter "retrigger_delay"
[2:44:00.340154075] [182360]  WARN RPiAf af.cpp:77 Missing parameter "check_for_ir"
[2:44:00.341007150] [182360]  INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@1000120000/rp1/i2c@80000/ov64a40@36 to CFE device /dev/media1 and ISP device /dev/media3 using PiSP variant BCM2712_C0
[2:44:00.354644124] [182351]  INFO Camera camera.cpp:1205 configuring streams: (0) 4624x3470-YUV420/sYCC
[2:44:00.356634978] [182360]  INFO RPI pisp.cpp:1483 Sensor: /base/axi/pcie@1000120000/rp1/i2c@88000/ov64a40@36 - Selected sensor format: 4624x3472-SRGGB10_1X10 - Selected CFE format: 4624x3472-PC1R

The final thing I can provide is our htop output, which shows that the processes are still running. This is the output we would see on each run of the rpicam-still command for both cameras, but obviously these PIDs are now stale.

182351 root        20   0  497M  134M  128M S   0.0  3.3  0:00.59 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182353 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.50 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182356 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182357 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182358 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182359 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182362 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182363 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182364 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182365 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182366 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182367 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182368 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182369 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182370 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182371 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182372 root        20   0  450M 89632 83488 S   0.0  2.2  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 1 -o /dev/shm/tmp1.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0
 182373 root        20   0  497M  134M  128M S   0.0  3.3  0:00.00 /usr/bin/rpicam-still --width 4626 --height 3472 --camera 0 -o /dev/shm/tmp0.jpg -n -t 3000 --no-raw --encoding jpg --quality 100 --rotation 0

Again, please let me know if I can provide any further information to help debugging this issue, and thanks for your help so far.

[naushir]

Can you run uname -a and post the output please?