posix: pthread: race condition between pthread_create() and pthread_join()

[hakehuang]

Previous Title
posix_common derived test fails on some qemu platforms when the CONFIG_TICKLESS_KERNEL=n is added

Describe the bug
posix_common derived test fails on some qemu platforms when the CONFIG_TICKLESS_KERNEL=n is added

Please also mention any information which could help others to understand
the problem you're facing:

• What target platform are you using?
  qemu_cortex_r5 and qemu_cortex_a53
• What have you tried to diagnose or workaround this issue?
  see discussion test_posix_common: fix test_clock_gettime_rollover failure #55208

1. add a delay in the test_pthread_descriptor_leak loop can help
2. use tickless mode can work as well

• Is this a regression? If yes, have you been able to "git bisect" it to a
  specific commit?
  No.

To Reproduce
Steps to reproduce the behavior:

1. add CONFIG_TICKLESS_KERNEL=n to prj.conf
2. do below

west build -b qemu_cortex_r5 -- -DCONFIG_NEWLIB_LIBC=y
west build -t run

Expected behavior
PASS

Impact
some potential race condition in pthread handling

Logs and console output

*** Booting Zephyr OS build zephyr-v3.3.0-1516-g1dce3c3ee2e9 ***
...
===================================================================
START - test_pthread_descriptor_leak

    Assertion failed at WEST_TOPDIR/zephyr/tests/posix/common/src/pthread.c:598: posix_apis_test_pthread_descriptor_leak: (pthread_join(pthread1, &unused) is non-zero)
unable to join thread 31
 FAIL - test_pthread_descriptor_leak in 0.032 seconds
===================================================================
START - test_sleep
ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/kernel/sched.c:1797
        aborted _current back from dead
E: r0/a1:  0x00000004  r1/a2:  0x00000705  r2/a3:  0xff00002c
E: r3/a4:  0x000089b9 r12/ip:  0x000242e8 r14/lr:  0x0000b2a1
E:  xpsr:  0x0000013f
E: Faulting instruction address (r15/pc): 0x0000c934
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Current thread: 0x20038 (unknown)
E: Halting system
...

Environment (please complete the following information):

• OS: (e.g. Linux, )
• Toolchain (e.g Zephyr SDK, ...)
• Commit SHA or Version used: zephyr-v3.3.0-1516-g1dce3c3ee2e9

[cfriedt]

On commit 336803e, I'm unable to reproduce this on main with the command below (also tried CONFIG_TICKLESS_KERNEL=n in prj.conf).

west build -p always -b qemu_cortex_a53 -t run tests/posix/common -- \
  -DCONFIG_TICKLESS_KERNEL=n -DCONFIG_NEWLIB_LIBC=y

Oddly, what I have found with qemu_cortex_r5 is that if I disable other tests within the same testsuite, the test passes. E.g.

for i in tests/posix/common/src/*.c; do
  j=$(basename $i)
  if [ "$j" = "main.c" -o $j = "pthread.c" ]; then
    continue
  fi
  k=$i.ignore
  mv $i $k
done
west build -p always -b qemu_cortex_r5 -t run tests/posix/common -- \
  -DCONFIG_TICKLESS_KERNEL=n -DCONFIG_NEWLIB_LIBC=y

I am able to reproduce this on qemu_cortex_r5 without disabling other parts of the testsuite though.

That leads me to believe we are looking at possible stack corruption (stack overflow, ..). Another alternative is that there may be some uninitialized variable.

[cfriedt]

I tried this [1], which sets stack sizes to a minimum size, but the problem persisted. That makes me think it's more of an uninitialized variable issue or maybe an alignment issue with the architecture.

If it's an uninitialized variable, it could take some time to pinpoint, unfortunately.

That's very odd that this issue only presents itself with qemu_cortex_r5 as well. @hakehuang - this happens with NXP hardware too, right?

@microbuilder, do you know if Cortex-R5 has any special requirements w.r.t. thread stack alignment?

[1]

git checkout .
west build -p always -b qemu_cortex_r5 tests/posix/common -- \
    -DCONFIG_TICKLESS_KERNEL=n -DCONFIG_NEWLIB_LIBC=y
  grep STACK build/zephyr/.config | \
  grep -v "^#" | \
  grep -v "=\(y\|=n\)$" | \
  while read -r LINE; do \
    VAL="$(echo "$LINE" | awk '{split($0,a,"="); print a[2];}')" \
    if [ $((2*VAL)) -lt 4096 ]; then \
      NEWVAL=8192 \
    else \
      NEWVAL=$((2*VAL)) \
    fi \
    CONFIG="$(echo "$LINE" | awk '{split($0,a,"="); print a[1];}')" \
    echo "$CONFIG=$NEWVAL" \
  done | \
  tee -a tests/posix/common/prj.conf
west build -p always -b qemu_cortex_r5 -t run tests/posix/common -- \
    -DCONFIG_TICKLESS_KERNEL=n -DCONFIG_NEWLIB_LIBC=y

[hakehuang]

That's very odd that this issue only presents itself with qemu_cortex_r5 as well. @hakehuang - this happens with NXP hardware too, right?

no, only in QEMU or renode platform

[hakehuang]

Oddly, what I have found with qemu_cortex_r5 is that if I disable other tests within the same testsuite, the test passes. E.g.

if the pthread test is removed, all other will not fail, and per my testing it is only the last test case in the pthread which will cause problem for itself or for following test, a simple way to check is add a sleep right after the test. which will trigger the problem.

[cfriedt]

@hakehuang @dleach02 - please take a look at #57403 - it seems the solution was a lot easier than we anticipated.