Can't catch exception of micropython code in C program

[bhcuong2008]

I have 3 test cases as below with MicroPython v1.21.0 on ESP32, also in my program.

1. Test case 1, I run this code, and the result as expected.

nlr_buf_t nlr;
if(nlr_push(&nlr) == 0) { 
  MP_CMOS_DICT = mp_load_global(MP_QSTR_CMOS);
  nlr_pop();
} else {
  mp_obj_print_exception(&mp_plat_print, (mp_obj_t)nlr.ret_val);
}

It prompts NameError: name 'CMOS' isn't defined as expected behavior in MicroPython.

2. Test case 2, the code is

MP_CMOS_DICT  = mp_const_none;
nlr_buf_t nlr;
if(nlr_push(&nlr) == 0) {
  mp_obj_t mp_fn = mp_load_attr(MP_CMOS_DICT, MP_QSTR_clear);
  mp_call_function_0(mp_fn);
  nlr_pop();
} else {
  mp_obj_print_exception(&mp_plat_print, (mp_obj_t)nlr.ret_val);
}

I can't catch python exception in C code. Exception decoder is:

0x401b4270: nlr_pop_jump_callback at micropython/py/nlr.c line 63
0x401b4297: nlr_call_jump_callbacks at micropython/py/nlr.c line 79
0x401b42bf: nlr_jump at micropython/py/nlrsetjmp.c line 32
0x401c0e93: mp_raise_msg_varg at micropython/py/runtime.c line 1676
0x401c11c9: mp_load_method at micropython/py/runtime.c line 1233
0x401c11d9: mp_load_attr at micropython/py/runtime.c line 1054

It can catch exception, but failed at nlr_pop_jump_callback , *top = (*top)->prev;, causing ESP32 exception and restart.

3. If I test in Micropython environment REPL,

CMOS = None
CMOS.clear 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'clear'

So, in micropython env REPL, it can catch exception as expected. I think test case 2 must be the same. It must prompt AttributeError: 'NoneType' object has no attribute 'clear', and not causing ESP exception LoadProhibited.

So what's wrong with my C code in test case 2?

Thanks,
Cuong.

[bhcuong2008]

I also test async code on MicroPython 1.21, running 2 async tasks, and get the same error as above. But with my old build on MicroPython 1.16 ESP32, it's ok (I also tested this code MP 1.16 in 2021, now I test it again).

#import uasyncio as asyncio # for 1.16
import asyncio
import _thread

async def _async_cb(params):
   print(params)   
   while True:
      print('Hello ' + str(params))
      await asyncio.sleep(params)

def testThread():
   asyncio.create_task(_async_cb(5))
   asyncio.create_task(_async_cb(10))
   loop = asyncio.get_event_loop()
   loop.run_forever()
 
_thread.start_new_thread(testThread, ())

[bhcuong2008]

if running directly in REPL main thread, it's ok. So I think that there is some issue in threading in MP 1.21. I will recheck my port.

[bhcuong2008]

I run test case 2 in the same main thread, then it promtps AttributeError: 'NoneType' object has no attribute 'clear'. So it must be thread-related issue.

MP_CMOS_DICT  = mp_const_none;
nlr_buf_t nlr;
if(nlr_push(&nlr) == 0) {
  mp_obj_t mp_fn = mp_load_attr(MP_CMOS_DICT, MP_QSTR_clear);
  mp_call_function_0(mp_fn);
  nlr_pop();
} else {
  mp_obj_print_exception(&mp_plat_print, (mp_obj_t)nlr.ret_val);
}

[bhcuong2008]

I run test case 2 in the same main thread, then it promtps AttributeError: 'NoneType' object has no attribute 'clear'. So it must be thread-related issue, especially exception catching. If there's no exception in other MP threads, then code is still running.

MP_CMOS_DICT  = mp_const_none;
nlr_buf_t nlr;
if(nlr_push(&nlr) == 0) {
  mp_obj_t mp_fn = mp_load_attr(MP_CMOS_DICT, MP_QSTR_clear);
  mp_call_function_0(mp_fn);
  nlr_pop();
} else {
  mp_obj_print_exception(&mp_plat_print, (mp_obj_t)nlr.ret_val);
}

[bhcuong2008]

I've checked threading port of ESP32 carefully, line by line. And run many tests in REPL main thread, and new mpy thread. I'm sure that this error caused by internal MicroPython in catching exception when running mpy code in other mpy threads.

[bhcuong2008]

In file py/nlrsetjmp.c MPY 1.21, I replace the code of function nlr_jump with the one from MPY 1.16, then it run fine.
Code from MPY 1.16:

void nlr_jump(void *val) {
    nlr_buf_t **top_ptr = &MP_STATE_THREAD(nlr_top);
    nlr_buf_t *top = *top_ptr;
    if (top == NULL) {
        nlr_jump_fail(val);
    }
    top->ret_val = val;
    MP_NLR_RESTORE_PYSTACK(top);
    *top_ptr = top->prev;
    longjmp(top->jmpbuf, 1);
}

And code from MPY 1.21:

void nlr_jump(void *val) {
    MP_NLR_JUMP_HEAD(val, top);
    longjmp(top->jmpbuf, 1);
}

The only difference is the call nlr_call_jump_callbacks(top); in MPY 1.21 (in macro MP_NLR_JUMP_HEAD).
nlr_call_jump_callbacks is not available in MPY 1.16.

// This function pops and runs all callbacks that were registered after `nlr`
// was pushed (via nlr_push).  It assumes:
//  - a descending C stack,
//  - that all nlr_jump_callback_node_t's in the linked-list pointed to by
//    nlr_jump_callback_top are on the C stack
// It works by popping each node in turn until the next node is NULL or above
// the `nlr` pointer on the C stack (and so pushed before `nlr` was pushed).
void nlr_call_jump_callbacks(nlr_buf_t *nlr) {
    nlr_jump_callback_node_t **top = &MP_STATE_THREAD(nlr_jump_callback_top);
    while (*top != NULL && (void *)*top < (void *)nlr) {
        nlr_pop_jump_callback(true);
    }
}

[jimmo]

Closing because this was also raised as #12695. I will reply there.