Using mp_get_buffer / mp_get_buffer_raise

[brianwyld]

Maybe someone can point me to the doc for these methods - I'm using them but wanted to check:

• what is the difference between them?
• who controls the buffer returned? Is it just a pointer to the micropython buffer, or a copy?
• what happens if I keep the reference after the method completes? How will I know if the buffer changes/gets garbage collected???

Same question for mp_obj_str_get_data : is it a copy or a reference and what happens after?

Right now I copy the buffer if I want to use the data afterwards, but just want to know if thats a waste of time/space....

thanks for any pointers.

[andrewleech]

The internal / C functions generally don't have separate docs, so it'd generally go straight to the source.

micropython/py/obj.c

Line 590 in 5473200

void mp_get_buffer_raise(mp_obj_t obj, mp_buffer_info_t *bufinfo, mp_uint_t flags) {

The mp_get_buffer returns a Boolean for success / fail, mp_get_buffer_raise simply wraps this and raises an exception on bool == false.

These functions don't copy data, they just give a C struct references to the data in the buffer.

If you keep a copy of the obj reference to the buffer that gets passed into these functions that should keep your original buffer alive, this is essentially holds the reference.

[jimmo]

* what is the difference between them?

As Andrew said, the _raise version will raise an exception for you if the buffer doesn't support the requested read/write mode.

* who controls the buffer returned? Is it just a pointer to the micropython buffer, or a copy?

It's owned by the object that gave you the buffer.

In general it's not safe to hold onto this if you're going to let other code run (i.e. return to Python).

If you're sure that the object won't change its buffer, then you can hold onto the pointer as long as you also ensure that the object won't be garbage collected.

If the object can change its buffer then you shouldn't keep the pointer, instead keep the mp_obj_t and use mp_get_buffer each time you need the buffer.

(i.e. in both cases, keeping a reference to the mp_obj_t is the safe thing to do).

* what happens if I keep the reference after the method completes? How will I know if the buffer changes/gets garbage collected???

I think this should be covered above -- you need to hold a reference to the mp_obj_t. This either needs to be via MP_REGISTER_ROOT_POINTER or from an object that has at least the lifetime of what you need.

See e.g. extmod/modframebuf.c where the mp_obj_framebuf_t holds both the bytearray object and the underlying buffer. So as long as something holds a reference to the framebuf, the backing bytearray is also kept live, even if the user code doesn't keep a reference to the bytearray directly.

Same question for mp_obj_str_get_data : is it a copy or a reference and what happens after?

Same as above. See #12404 (comment) where I forgot to take this into account recently.

Right now I copy the buffer if I want to use the data afterwards, but just want to know if thats a waste of time/space....

With strings, they can be interned in which case the buffer is totally fine to hold on to. But of they're not interned then the string object might be gc'ed. But in either case strings are immutable, so as long as you hold onto the mp_obj_t you're fine.

thanks for any pointers.

[jimmo]

If you're sure that the object won't change its buffer, then you can hold onto the pointer as long as you also ensure that the object won't be garbage collected.

If the object can change its buffer then you shouldn't keep the pointer, instead keep the mp_obj_t and use mp_get_buffer each time you need the buffer.

Note that the framebuf example is making the assumption that the buffer-providing object (i.e. bytearray) won't change.

If you resize the bytearray in-use by a framebuf then...yeah...bad times. But this should never be necessary. framebuf could guard against this by always using mp_get_buffer in every operation but that would add a (small) performance hit.

[jimmo]

Also I said "reference" several times when talking about the GC, but important to note that micropython is not reference-counted. By "reference" i really mean "findable from a root pointer".

Sorry there aren't detailed written docs on this, but the best I can offer is https://youtu.be/H_xq8IYjh2w

[glenn20]

Note that the framebuf example is making the assumption that the buffer-providing object (i.e. bytearray) won't change.

If you resize the bytearray in-use by a framebuf then...yeah...bad times. But this should never be necessary. framebuf could guard against this by always using mp_get_buffer in every operation but that would add a (small) performance hit.

Got me thinking... would a solution to this be to create an internal memoryview object referencing the bytearray storage (and add it to a root pointers protected object)?

[jimmo]

@brianwyld @glenn20 I should add an important clarification to what I wrote above.

For garbage collection, a given allocation will be kept alive as long as you can find a pointer to the start of that allocation starting at some root pointer.

So a bytearray for example is an object that contains a pointer to an allocation that holds the actual bytes data.

If you call mp_get_buffer on that bytearray, you get back the pointer to the bytearray's data allocation. So if you now hold onto that pointer, it's actually safe to let the bytearray object get collected, because you are now keeping the actual data allocation live.

The problem is, not all objects that implement the buffer protocol (i.e. the thing that makes then support mp_get_buffer) will return a pointer that is the start of their allocation, and so the pointer you get is not necessarily . It just so happens that bytearray does, but this is not true in general. This is why you also need to hold onto the underlying object.

memoryview is an interesting one in that it really assumes it's working with a bytearray, as it does only hold onto the allocation. This is all described in a long comment here: https://github.com/micropython/micropython/blob/master/py/objarray.c#L39

would a solution to this be to create an internal memoryview object referencing the bytearray storage

So memoryview won't actually solve the problem here as it itself has the same problem.

[brianwyld]

thanks a lot guys, that gives me a much better picture of what's going on and what I can (and can't) do....