Native machine code: passing bytearrays

[DavideRossi]

I would like to use a native C function (see https://docs.micropython.org/en/latest/develop/natmod.html) to process a Python bytearray. I also checked the examples under examples/natmod/ but I only found how to deal with other types. Do you have some examples to share? Thanks.

[rkompass]

Native C functions use the same collection of functions and objects that is used by Micropython internally.
So you may search the MP code for similar uses to extract the functions needed.
Unfortunately these functions are not documented. Fortunately they are quite clearly coded and well understandable.
I had a look at py/objarray.c:
There the function static void array_print(const mp_print_t *print, mp_obj_t o_in, mp_print_kind_t kind) seems to give an example how to access the bytearray.
Assuming the bytearray is given as MP object so: mp_obj_t o_in
There is mp_obj_array_t *o = MP_OBJ_TO_PTR(o_in); to create the structure holding the essential informations for it's access.
o->items and o->len seem to be the pointer to the array contents and the lenght of the array.
o->typecode == BYTEARRAY_TYPECODE might be used as test wether the array truely is a bytearray.

[jonnor]

bytearray is implemented same way as array.array with typecode='B'.
So yes, you do the typecode checking, and then you can cast the data pointer to an uint8_t, and then access as a C array/pointer.
Example:
https://github.com/emlearn/emlearn-micropython/blob/b8c2eb3b0f34ca2014a6e536a40a6451f2ae02f8/src/emlearn_kmeans/kmeans.c#L52

[DavideRossi]

Thanks both @rkompass and @jonnor.
I'm summarizing here the solutions for future readers.
Solution 1:

static func(mp_obj_t object_in) {
  mp_obj_array_t *array_obj = MP_OBJ_TO_PTR(object_in);
  if (array_obj->typecode != BYTEARRAY_TYPECODE) {
    mp_raise_ValueError("expected a bytearray");
  }
  uint8_t *values = (uint8_t *)array_obj->items;

Solution 2:

static func(mp_obj_t object_in) {
  mp_buffer_info_t bufinfo;
  mp_get_buffer_raise(object_in, &bufinfo, MP_BUFFER_RW);
  if (bufinfo.typecode != BYTEARRAY_TYPECODE) {
    mp_raise_ValueError("expected a bytearray");
  }
  uint8_t *values = bufinfo.buf;

Also for future readers, I implemented and benchmarked a silly function that shuffles some bits around in the bytearray in Python, Pyton+Viper and C.
These are the results for an ESP32-S3:

	Python	Python+Viper	C
Time (microseconds)	728662	25080	2242
Speedup	1x	29x	325x

To summarize: Python is great for prototyping and when no number-crunching is needed (which, I guess, covers a lot of use cases). But if you need to squeeze some performance out of these microcontrollers you don't really have a decent language supporting async. Unlucky the last time I checked, Rust was still a no go (I don't even remember what the show stopper was...).

[jonnor]

@DavideRossi very nice! For reference, how long was the bytearray that you benchmarked?

One can use a C module in MicroPython in combination with async. So that still seems to be a viable language choice.

[DavideRossi]

I kind of lost track of the exact size (and even the specific algorithm) I was using for that test, I kept changing the code and trying different stuff. But I believe that was a 50000 items bytearray.
And yes, I can mix Python and C, and this in fact is what I'll be doing, but I would really prefer a one-language solution. Also because the build process in quite a pain at this point...