Locking function when executed from micropython

[ghost]

Hi there,

I am facing a problem and I want to better understand what's happening.

I have created a new module and imported into my micropython project. This module is interfacing to a device via UART.
This module implements a task that manages the UART messages. The task is created with the xTaskCreatePinnedToCore to the CORE 1 (the same where micropython should be running) and priority is 7.

There are some more functions that are part of the module and that I've mapped to Micropython.
Everything is working correctly, I mean the module is called, the task is created and I'm able to see some log that confirm that everything is working correctly.

The problem is that when I try to call one of the mapped function the task seems to be Locked and I am not able to do anything on the REPL until the called function finishes to run.
I am not able to understand why this function is not locking when executed from the task itself but becomes locking when executed from micropython.
Can you please suggest to me what I'm doing wrong?

Thank you very much!
Greetings,
Andrea

[peterhinch]

I don't entirely understand what you are doing, but it seems you are trying to implement concurrency on an ESP32. There are two ways to achieve this in MicroPython: one is threading using the _thread module. This should only be used when absolutely necessary for reasons discussed here.

In most cases, and certainly for tasks like handling a UART, the answer is uasyncio. See docs and tutorial. There is an example UART script here.

[glenn20]

We can only speculate as to the problem from the limited information. From your message, I assume:

• You have written a module in C (you use xTaskXreatePinnedToCore) and made various functions available to be called from micropython.
• When you load your module, you start a thread which monitors the UART (the UART thread)
  • and presumably exchanges messages/data with the micropython thread in some manner
• You invoke some function from python which is also used by your UART thread.
• When you call it from python it takes some time to execute before returning to the repl, which is not what you expected.
• When your UART thread invokes the function it executes quickly.

I can imagine this could be caused by some locking of a resource that is being held by your UART thread which eventually releases it, enabling the python thread to complete.

Or possibly, you are misundertanding the nature of threading. Eg, rather than call the function from python, you should instead send a message to the UART thread to execute the function for you.

It is really hard to help without more information.

Also - how did you ascertain that your UART thread is "Locked"?

[ghost]

Hi @glenn20 thank you for your support.

I can try to better explain what I'm doing.

You have written a module in C (you use xTaskXreatePinnedToCore) and made various functions available to be called from micropython.

That's correct.

When you load your module, you start a thread which monitors the UART (the UART thread)
and presumably exchanges messages/data with the micropython thread in some manner

That's correct. The module itself runs a Task that manages the UART outcoming and incoming messages. Messages are appended into a Queue. To get those messages into my micropython app I've writted a read" function that is mapped from c to micropython. This function gets those messages from the queue

You invoke some function from python which is also used by your UART thread.

Almost correct. I have a function written in C (let's say "is_it_ok") code that sends a command on the UART and waits for the answer. Once I get the answer i change the state of my FSM depending on what I get.
I've added another function that is mapped to micropython (let's say "upy_is_it_ok") and that recalls the previous function "is_it_ok".
The two functions are never called together!

When you call it from python it takes some time to execute before returning to the repl, which is not what you expected.
When your UART thread invokes the function it executes quickly.

No, it's not properly like that. It's not a time problem. What I see is that when the function is executed directly from the module tasks into c code I can observe the context switching from one task to onother while I'm waiting the answer from the UART. So nothing bad here.

In my micropython application I've a manager of this module, from where I call the mapped C functions depending on the FSM state. This manager starts a _thread.start_new_thread() that calls the "upy_is_it_ok" function. Just to remember, this function is mapped to the same function that's called from the module task itself "is_it_ok".

The difference appears here. In this case I don't have any context switching while I'm waiting for the UART response, unlike what happens into the module task when the c function is called.

In short:
xTaskXreatePinnedToCore - > C task -> C function "is_it_ok" -> Context Switching -> other tasks continue to run while I wait for UART response
_thread.start_new_thread() - > micropython task -> micropy function "upy_is_it_ok" -> No Context Switching -> other tasks are locked until I get UART data

[glenn20]

This behaviour MAY be caused by the micropython thread that is invoking upy_is_it_ok not releasing the micropython GIL while it is waiting for data from the UART (hard to be sure). Note that your UART monitor is a thread/task that does not participate in the locking of the GIL (it is not a python task), so it will not block other tasks when it executes is_it_ok.

If this is the case, then your code wil either perform a blocking operation till data is available OR enter a busy loop checking for data to be available. You should either release the GIL with MP_THREAD_GIL_EXIT() before blocking (and take it back before continuing with MP_THREAD_GIL_ENTER()) or call the MICROPY_EVENT_POLL_HOOK macro (or call mp_hal_delay_ms(ms)) in your loop to allow other micropython tasks to take the GIL while you are waiting. You will need a mechanism to do that only if being called from a micropython thread and not a non-micropython thread.

(I am going to qualify all this by saying I am very inexpert with micropython threading, just based on what I expect should be happening. Good luck).

[ghost]

Dear @glenn20 , I've added the MP_THREAD_GIL_EXIT() and MP_THREAD_GIL_ENTER() but it seems that nothing changes but I'm quite sure that you gave me a great help and some ideas. I'll make some more tests and code changements and I'll let you know.

Thank you very much for your help!

[glenn20]

Sorry it didn't fix your problem, though I suspect the issue is something like I have described. Also - remember that, however you do it, once a task releases the GIL, it should not change the state of any python objects till it gets the GIL back.