Micropython RuntimeError: maximum recursion depth exceeded

[pollardd]

My project needs part of it to run in a second thread so that the user interface remains responsive. When I run my program completely in the main thread it works no problem it's just less responsive.
When I move part of the code into a second thread, I receive the recursion depth exceeded error when a particular part of the code is triggered. There is no recursive code in my project. I think the available resources are just less in the second thread. It appears from the error below that the function calls are about 11 levels deep when the error occurs.

Here is the Micropython Veersion and Pico Board
MicroPython v1.19.1-746-gf2de289ef on 2022-12-13; Raspberry Pi Pico W with RP2040

Here is the error I'm receiving.

Traceback (most recent call last):
  File "sx1262.py", line 275, in _onIRQ
  File "subprocess.py", line 73, in cb
  File "sx1262.py", line 187, in recv
  File "sx1262.py", line 251, in _readData
  File "sx126x.py", line 483, in startReceive
  File "sx126x.py", line 540, in startReceiveCommon
  File "sx126x.py", line 1133, in setPacketParams
  File "sx126x.py", line 1228, in fixInvertedIQ
  File "sx126x.py", line 1034, in writeRegister
  File "sx126x.py", line 1274, in SPIwriteCommand
  File "sx126x.py", line 1291, in SPItransfer
RuntimeError: maximum recursion depth exceeded

What I think I need to do is increase the Stack Size for the second thread.
If people really want to see the code I can go public with my github repository but the code is fairly complicated and I don't want to scare people away :)

I have come up with the following example to trigger the same error message using recursion.
The function can run in the second thread and call itself 17 times before it fails.
The function can call itself 38 times in the main thread which is why I think there are less resources available to the second thread.

I'm hoping to be able to increase the Stack Size and allow more calls before it fails.
Changing the stack size with "_thread.stack_size()" in my test below doesn't make any difference.

Any ideas how I can allow more resources to allow my code to run?

# Based on an example found here
# https://forum.micropython.org/viewtopic.php?t=3091

# Stack Size details found here (for regular python)
# https://superfastpython.com/thread-stack-size-in-python/

import time
import _thread 
a = 0
fail = False

oldStackSize = _thread.stack_size(16 * 4096)
print("Old Stack Size="+str(oldStackSize))

newStackSize = _thread.stack_size()
print("New Stack Size="+ str(newStackSize))
    
def recursionTest():
    global a, fail
    a += 1
    print("loop count="+str(a))
    if not fail:
        try:
            recursionTest()
        except Exception as errorMsg:
            print(errorMsg)
            fail = True

# Runs in the primary thread
#print("Main thread")
#recursionTest()

# Runs in the second thread
print("Sub thread")
_thread.start_new_thread(recursionTest,())

UPDATE

I created 8000 byte string within the recursion loop and the process failed after 15 loops
Failed with a different error though. So the number of recursions is also affected by other factors not just the depth.

loop count=13
8000
loop count=14
8000
loop count=15
memory allocation failed, allocating 8001 bytes

[davefes]

Probably not much help, but when I had that error the author told me to "flatten it", maybe:
https://stackoverflow.com/questions/12472338/flattening-a-list-recursively

At which point I sensed I was totally out of my depth!

[pollardd]

@davefes,
Yes I have that idea in the back of my head. The code with all the layered function calls is from a third party.
It controls a Pico Module from Waveshare named "Pico-LoRa-SX126x"
As the name suggests it allows for LoRa radio communications between Pico modules over large distances.
I could duplicate code within the functions to reduce the number of levels without really understanding what it's doing exactly. I was hopping I could just increase some memory size and leave that code alone.
We shall see :)
I like your analogy of being in too deep. :) Your list problem looked very confusing.
David

[jimmo]

@pollardd

There is no recursive code in my project.

The error message is slightly confusing -- inherited from Python where the most likely cause of running out of stack space is due to recursion. But yes, in this case there's just a limit of how deep the call stack can be.

The second core does get a smaller stack (4k by default).

To set the stack size for the second core, use _thread.stack_size(n_bytes) before calling _thread.start_new_thread.

That said, I think there might be a bug with the way the stakc check works for the second core. I will investigate and follow up. (Edit: This seems to work as expected)

UPDATE I created 8000 byte string within the recursion loop and the process failed after 15 loops

This is failing due to running out of heap, not stack.

The code with all the layered function calls is from a third party.

Indeed that's quite a deep call stack.

We're working on implementing a driver for this module in micropython-lib. Maybe you could give that a go. See micropython/micropython-lib#501 (comment)

[pollardd]

@rkompass,
I never thought that no arguments would silently set the stack size back to the default 4k.
I read an article (didn't keep the link) that went into a lot of details on this command but either it didn't mention this behavior or I missed it.
They seem to have over complicated the way this command works.
As you say the obvious behavior would simply be.
Set the stack size _thread.stack_size(8*1024)
Report the stack size setting size= _thread.stack_size()
With an error if something fails.
Isn't that the way life should be :)

This certainly wasted a lot of my time but I'm glad with some help from you guys we got to the bottom of it.
Thanks
David

[jimmo]

@rkompass @pollardd Just to clarify, this is (confusingly) the correct behavior of the _thread.stack_size() function. See https://docs.python.org/3/library/_thread.html#thread.stack_size

[karfas]

Return the thread stack size used when creating new threads. The optional size argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive integer value of at least 32,768 (32 KiB). If size is not specified, 0 is used. If changing the thread stack size is unsupported, a RuntimeError is raised. If the specified stack size is invalid, a ValueError is raised and the stack size is unmodified.

@jimmo: I have no idea how this behaves in CPython, but I don't read the library docs in a way that an unspecified size parameter resets the stack size. In "The optional size argument ... must be be 0 (use platform or configured default) ..." could also mean simply to return the previous (by stack_size() ?) configured (default ?) value.
As the first sentence states "Return the thread stack size used when creating new threads" then it should at least return zero when it sets the stack size to zero...

However, I'm not a native English speaker, so I might miss important nuances.

The current code in py/modthread.c does reset the stack size, but does not "Return the thread stack size used when creating new threads".

Bug ?

STATIC size_t thread_stack_size = 0;
/* ... unrelated stuff ... 
 */
STATIC mp_obj_t mod_thread_stack_size(size_t n_args, const mp_obj_t *args) {
    mp_obj_t ret = mp_obj_new_int_from_uint(thread_stack_size);
    if (n_args == 0) {
        thread_stack_size = 0;
    } else {
        thread_stack_size = mp_obj_get_int(args[0]);
    }
    return ret;
}

[rkompass]

Thanks @karfas, I hadn't the endurance to look into the sources, (also I'm not used to produce pull requests anyway), but this looks like it should be:
return thread_stack_size;
in the end.
Still It would feel better if instead of 0 the value 4096 would be returned.
....
Now I used my test program with ordinary python (3.10.9) and it behaves like MP: without argument the stack size is reset and the return value always is the previous stack size. Instead of the default value always a 0 is returned. So MPs behaviour is identical at least, as jimmo stated.
Scratching my head I remember that _thread is only a minimal set of functionality that in Pythons threading module is used to get a much larger set of functionalities with more intuitive behavior - at the cost of much overhead, several locks for example iirc for a simple task. Something that does not feel suitable for MP.
How about:

def xstack_size(ss=None):
    if ss is not None:
        _thread.stack_size(ss)
        r = ss
    else:
        r = _thread.stack_size()
        _thread.stack_size(r)
        if r == 0:
            r = 4096
    return r

[jimmo]

@karfas The wording is very subtle, bordering on ambiguous, and it all hinges around the use of the word "subsequently" in the second sentence. They could probably improve it marginally by adding "currently" to the first sentence.

But just to be really clear, this is indeed how CPython works.

$ python
Python 3.10.9 (main, Dec 19 2022, 17:35:49) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _thread
>>> _thread.stack_size()
0
>>> _thread.stack_size(2*1024**2)
0
>>> _thread.stack_size(4*1024**2)
2097152
>>> _thread.stack_size(0)
4194304
>>> _thread.stack_size()
0
>>> _thread.stack_size(2*1024**2)
0
>>> _thread.stack_size()
2097152