mp_sched_schedule with multiple args or tuple from a different thread / task

[bwhitman]

I am using mp_sched_schedule to call a Python user callback in C code for Tulip. . The c code necessarily happens in a different thread (and on hardware, ESP FreeRTOS task). I want to pass two integers to the users' callback. I'm finding that sending one value in the callback works fine, like:

mp_sched_schedule(sequencer_callback, mp_obj_new_int(value));

The user can write

def cb(x):
    print("value is %d" % (x))

But i want to send two values. mp_sched_schedule only allows one argument. So I tried making a tuple in the C code:

mp_obj_t tuple[2];
tuple[0] = mp_obj_new_int(value1);
tuple[1] = mp_obj_new_int(value2);
mp_sched_schedule(sequencer_callback, mp_obj_new_tuple(2, tuple));

This crashes when creating the tuple with

0   tulip                         	       0x104fc8d10 gc_alloc + 68 (gc.c:600)
1   tulip                         	       0x104fc8278 m_malloc + 28 (malloc.c:86)
2   tulip                         	       0x1050b2ed0 mp_obj_malloc_helper + 28 (obj.c:42)
3   tulip                         	       0x1051ba928 mp_obj_new_tuple + 84 (objtuple.c:247)

Presumably, trying to access the MP GC from another thread is bad. (But i'm unsure why doing a single mp_obj_new_int works reliably.)

Anyway, I want to send two values from C to a Python function from another thread. What can i do?

[dlech]

mp_obj_new_int() will not allocate as long as the value fits in a small int so that explains why it doesn't trigger the crash. But yeah, most MicroPython APIs are not safe to call in this context.

So I guess you would either need to grab the MicroPython GIL or preallocate an int array to use here.

[bwhitman]

Ah, thanks for the explanation. Can you explain "Grab the micropython GIL" a bit, what would that look like? I can't really pre-allocate for this use case.

[dlech]

what would that look like?

MP_THREAD_GIL_ENTER();

// do MicroPython stuff (that doesn't use thread state because there is no current MicroPython thread)

MP_THREAD_GIL_EXIT();

Not sure if you can actually get away with this though. Never tried it.

[glenn20]

Are you sure you can"t preallocate the tuple?

Better to use a preallocated list as technically you are mutating the contents, but it will still work with a tuple. And then just fill the tuple with ints from your thread using: list->items[0] = mp_obj_new_int(value1); list->items[1] = mp_obj_new_int(value2);

However, you will be overwriting the contents every time the thread writes new data and some data may be lost if the callback does not run in time before the next message (ie. the callback will always see the latest data, not necessarily the data intended for it.

If you want to mitigate this, then yoiu need to think of your task as sending a stream of event data between threads which gets morte complicated. Micropython has a ringbuffer implementation in py/ringbuf.c that you could use for this - but things are getting more complicated.

[bwhitman]

Yeah, it's a fast stream of callback data (say, 5-10Hz) and in my experience with it, the scheduled callbacks are not always consumed spaced properly apart. Sometimes two are called "at once". Since these two numbers I want to send are mutable timestamps and indices to the event, if two happen "at once" I want the first one to represent t0 and the second one to represent t1 so that the python CB can do the right thing with the timestamps of each message.

But I'm thinking i could just make my own ringbuf instead, of, say 100 items in a list and just cycle through those pre-allocated lists when I'm sending. If the callbacks were behind by that many messages, that's another problem :)

[glenn20]

Just remember to ensure your ringbuf implementation is threadsafe (or threadsafe-enough for your application). You will be mutating state of python objects from witin your thread, while that data is being used by micropython. py/ringbuf.c is threadsafe for one thread writing and another one reading.

A possible design choice is to just write your data events from your thread into an internal "ringbuffer" (however implemented) and provide a function (or method) callable from python in your C module to read the next event data from the ringbuffer (returning it as a tuple of ints or whatever suits). You don"t need callbacks then - you can just poll the ringbuffer. You could still support callbacks if you want. The callback just calls the function to read from the internal ringbuffer, rather than receiving the data as arguments.

If the callbacks were behind by that many messages, that's another problem :)

Also, the micropython scheduler stack is limited to 8 entries by default, so if you are getting that far behind, you have already lost with the scheduler approach.

[bwhitman]

MP_THREAD_GIL_ENTER();

// do MicroPython stuff (that doesn't use thread state because there is no current MicroPython thread)

MP_THREAD_GIL_EXIT();

I did just give this a try and get the same crash during tuple creation.

[glenn20]

Interesting. It was def worth a try, but as @dlech suggested, it may not be possible in micropython on esp32. Micropython code on esp32 is designed to be run on one thread.

[jimmo]

@bwhitman Without pre-allocating, the only way to solve this is to run the Python callback synchronously. This is what we do with Bluetooth on ESP32. Right now this is not generic and is very tied to the Bluetooth implementation (search for MICROPY_PY_BLUETOOTH_USE_SYNC_EVENTS_WITH_INTERLOCK in extmod/modbluetooth.c). The way this works is that the freertos thread acquires the GIL (to lock out any MicroPython threads), and then takes over the thread state of the main MicroPython thread, and then invokes the callback.

@dlech @glenn20 Acquiring the GIL on its own is not sufficient because the GC's allocator access thread state.

We are looking at making this a generic function -- i.e. "run_on_micropython_thread" that provides a similar API to mp_sched_schedule. I have already implemented this for GIL and non-GIL builds, but still ironing out some details.

[glenn20]

That sounds interesting. I expect it would make it a lot easier to utilise threaded c code.

Acquiring the GIL on its own is not sufficient because the GC's allocator access thread state.

Yeah, that is what I expected. Good to have it confirmed.