How to pass micropython objects from arbitrary Freertos thread in C when calling a python function

[moefear85]

Hi,

I have an arbitrary freertos task in C (on esp32, using esp-idf). This task has a handle to to a micropython callback, let's call it mpy_callback (as an instance of mp_obj_t). I'm able to trigger it using mp_sched_schedule(mpy_callback,mp_const_none). But if I create any datums, such as mp_obj_new_int(...) or mp_obj_new_bytes(...), I get a GuruMeditation, even if I don't actually reference those datums or pass them to mpy_callback. What would be the correct way to pass information to the callback? Ideally I don't want anything global/persistent, but if it is necessary to do so, or if root pointers are involved (not sure what they are or how to create/use them), what would be the right way?

I've seen #10188 but there was no answer, only something along the lines of "you should research", which imo is never an answer, rather stating the obvious, like asking who does this raincoat belong to and someone answering: "to its owner". Asking a question in a forum IS research, so it would be nice if someone who knows would simply answer, rather than say "you should find the answer" (everybody knows that).

I've looked into machine_pin.h for the esp32 port. It's cryptic and doesn't answer the question. It is unclear where/how arguments are being created & given to the called micropython function.

[dlech]

my_callback will need to be reachable by the GC somehow to avoid releasing the memory too early. This could be done by creating a new root pointer (MP_REGISTER_ROOT_POINTER()) or by attaching it to some other Python object whose lifetime is synchronized with the rtos task.

All of the ISR handling rules apply here. So instead of mp_obj_new_int(), you could use MP_OBJ_NEW_SMALL_INT() (as long as you don't need all 32 bits) and for bytes, you would need to use a pre-allocated bytearray.

[moefear85]

thx for several hints that I haven't seen mentioned elsewhere. I'll look into them. Currently I'm trying the workaround of having the freertos task call a python internal/private callback without any parameters using mp_sched_schedule, then that callback calls back into C, creates/fills whatever micropython values needed/necessary, takes the results as a return tuple, then itself calls the user-facing callback giving it the datums as arguments. It's working up til now.

[jimmo]

@moefear85 In #8945 I am working on a way for a RTOS task to synchronously call into MicroPython to execute a callback. We need this for BLE events, and currently there's a specific implementation of this hidden inside modbluetooth.c, so this PR makes it into a more generic utility. The PR is WIP though and still being tested so I wouldn't rely on it for now, but this is something we hope to merge soon.

[moefear85]

@jimmo

does that mean it is unsafe for some reason for an ISR like the espnow receive callback to directly call mp_sched_schedule? If unsafe, does it become safe if the ISR messages a separate low priority task that in turn calls mp_sched_schedule? Does such a separate task have to be priority 0, or what highest priority would not interfere with mpy? If all are unsafe, what is the current safe way for an ISR to ultimately pass information to an mpy function?

[jimmo]

@moefear85 It's always safe to call mp_sched_schedule.

The issue is that whatever you pass as the argument to schedule needs to be still valid when the scheduler runs.

By running the callback synchronously (as proposed in #8945) then you don't have this problem.

[moefear85]

ok. I'm not facing such difficulties as freertos provides queues to pass information across threads without issues. I just mp_sched_schedule(function,mp_const_none) to trigger the desired python function, which then calls into C, pops off the needed information off the queue, then returns.