[BUG] Failing QEMU tests after #16499

[fdcavalcanti]

Description / Steps to reproduce the issue

Hi folks,

I've tracked commit b8e30b5 (fs/vfs: Separate file descriptors from file descriptions) from #16499 to an issue on Espressif's CI.

Our QEMU tests on ESP32-C3 started failing on "ps" command.

Here's where it stops:

[...]
Attempting to boot anyway...
entry 0x4038383c
*** Booting NuttX ***
dram: lma 0x00000020 vma 0x3fc83c10 len 0x788    (1928)
iram: lma 0x000007b0 vma 0x40380000 len 0x3c04   (15364)
padd: lma 0x000043c8 vma 0x00000000 len 0xbc30   (48176)
imap: lma 0x00010000 vma 0x42000000 len 0x1ee54  (126548)
padd: lma 0x0002ee5c vma 0x00000000 len 0x119c   (4508)
dmap: lma 0x00030000 vma 0x3c020000 len 0xbce0   (48352)
total segments stored 6
rtc_init(dbg): not burn core voltage in efuse or burn wrong voltage value in chip version: 03
ABC
NuttShell (NSH) NuttX-10.4.0
nsh> ps
  PID GROUP PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK COMMAND
    0     0   0 FIFO     Kthread   - Ready              0000000000000000 0002032 Idle_Task
    1     1 100 RR       Task      - Running            0000000000000000

I suspect something related to DEBUG_ASSERTIONS, but was not able to dig deeper.

And to reproduce:

• Checkout to said commit
• ./tools/configure.sh esp32c3-generic:ostest
• Enable: CONFIG_ESPRESSIF_MERGE_BINS, CONFIG_DEBUG_FEATURES, CONFIG_DEBUG_ASSERTIONS
• make
• Run QEMU with: qemu-system-riscv32 -nographic -icount 3 -machine esp32c3 -drive file=nuttx.merged.bin,if=mtd,format=raw
• Call ps

Can we take a look at this?
I'm available to run any tests required.

Thanks!

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Mint 22.1

NuttX Version

master

Issue Architecture

[Arch: risc-v]

Issue Area

[Area: Debugging]

Host information

No response

Verification

• I have verified before submitting the report.

[acassis]

@pussuw please take a look

[pussuw]

Sorry for the regression. Maybe @Donny9 can help you. He refactored big chunks of that patch and did the testing afterwards.

I don't have time for NuttX anymore as I will very shortly move to a different line of work, and will stop contributing to NuttX in any shape or form (at least for the foreseeable future).

[acassis]

@pussuw that is a sad news for our community! You became a very important contributor, not only for RISC-V/MPFS but also for other features. But that is part of life, sometimes we need to work with things we don't like just to pay the bill. :'(

[xiaoxiang781216]

@Donny9 is looking the problem.

[michallenc]

I can confirm this was happening on a real hardware as well (and right after the the boot in my configuration). Based on a gdb backtrace, the issue was in vfs/fs_close.c:146 double call to lib_put_tempbuffer (thus double free) if inotify was enabled.

Breakpoint 2, lib_put_tempbuffer (buffer=0x20400eb4 <g_tempbuffer+4> "tmpc000001") at misc/lib_tempbuffer.c:137
137	{
(gdb) bt
#0  lib_put_tempbuffer (buffer=0x20400eb4 <g_tempbuffer+4> "tmpc000001") at misc/lib_tempbuffer.c:137
#1  0x0043749c in file_close (filep=0x20419808) at vfs/fs_close.c:136
#2  0x00436d5a in file_put (filep=0x20419808) at inode/fs_files.c:897
#3  file_put (filep=0x20419808) at /home/michal/Michal/Elektroline/jessie/nuttx/brc_upstream/nuttx/core/include/nuttx/fs/fs.h:1309
#4  0x00437502 in close (fd=fd@entry=3) at vfs/fs_close.c:218
#5  0x0044522c in issmartfs (pathname=0x20414d88 "/dev/smart0p2") at board/sam_w25qxxxjv.c:154
#6  sam_w25qxxxjv_init () at board/sam_w25qxxxjv.c:399
#7  0x00444ad4 in sam_bringup () at board/sam_bringup.c:140
#8  0x00444a46 in board_app_initialize (arg=arg@entry=0) at board/sam_appinit.c:89
#9  0x0045061c in boardctl (cmd=cmd@entry=65281, arg=arg@entry=0) at boardctl.c:370
#10 0x0042f164 in nsh_initialize () at nsh_init.c:149
#11 0x0042f024 in startup_main (argc=1, argv=0x20413ea0) at startup_main.c:127
#12 0x00428b0e in nxtask_startup (entrypt=0x42f005 <startup_main>, argc=1, argv=0x20413ea0) at sched/task_startup.c:72
#13 0x004249f2 in nxtask_start () at task/task_start.c:104
#14 0x00000000 in ?? ()
(gdb) c
Continuing.
halted: PC: 0x0042c19c

Breakpoint 2, lib_put_tempbuffer (buffer=0x20400eb4 <g_tempbuffer+4> "tmpc000001") at misc/lib_tempbuffer.c:137
137	{
(gdb) bt
#0  lib_put_tempbuffer (buffer=0x20400eb4 <g_tempbuffer+4> "tmpc000001") at misc/lib_tempbuffer.c:137
#1  0x004374aa in file_close (filep=0x20419808) at vfs/fs_close.c:146
#2  0x00436d5a in file_put (filep=0x20419808) at inode/fs_files.c:897
#3  file_put (filep=0x20419808) at /home/michal/Michal/Elektroline/jessie/nuttx/brc_upstream/nuttx/core/include/nuttx/fs/fs.h:1309
#4  0x00437502 in close (fd=fd@entry=3) at vfs/fs_close.c:218
#5  0x0044522c in issmartfs (pathname=0x20414d88 "/dev/smart0p2") at board/sam_w25qxxxjv.c:154
#6  sam_w25qxxxjv_init () at board/sam_w25qxxxjv.c:399
#7  0x00444ad4 in sam_bringup () at board/sam_bringup.c:140
#8  0x00444a46 in board_app_initialize (arg=arg@entry=0) at board/sam_appinit.c:89
#9  0x0045061c in boardctl (cmd=cmd@entry=65281, arg=arg@entry=0) at boardctl.c:370
#10 0x0042f164 in nsh_initialize () at nsh_init.c:149
#11 0x0042f024 in startup_main (argc=1, argv=0x20413ea0) at startup_main.c:127
#12 0x00428b0e in nxtask_startup (entrypt=0x42f005 <startup_main>, argc=1, argv=0x20413ea0) at sched/task_startup.c:72
#13 0x004249f2 in nxtask_start () at task/task_start.c:104
#14 0x00000000 in ?? ()
(gdb) c

This change 16538 should solve the issue. At least did in my case.

[fdcavalcanti]

It works! Thank you @michallenc.

[fdcavalcanti]

@xiaoxiang781216 @michallenc
Folks, sorry for reopening this, but seems issue still persists.

When I tested, I had some memory debug functions enabled and seems to have masked the problem.
I ran the internal CI and noticed that half the tests still fail.
Same issue, happens on QEMU but not on hardware.