Config headers for dynamic native modules?

[dubiousjim]

I've been experimenting with dynamic native modules (as in the examples/natmod folders, building .mpy files with native code contents). I'm puzzled by the fact that the examples, although they build and run fine, end up using different config macros than the port (Unix/standard variant) I'm loading them into. The MICROPY_CONFIG_ROM_LEVEL for example is different, and many other settings. Not only does this end up hiding some parts of the C API from my native module, it also changes the definitions of some structs and so changes the ABI. It's just lucky then that the examples I've built haven't crashed.

I expect that a proper build process would first incorporate all the macro definitions in my mpconfigport.mk and mpconfigport.h (and their includes), then overlay any changes appropriate to the dynamic module compilation. It doesn't seem that the first of these steps is handled in any of the examples.

Am I understanding things right? How do others manage this?

[jonnor]

Natmod builds are (should be) completely separate from any particular port or port configuration. Basically the entire C API should be hidden - only a very limited set of functions are available - via the symbol table - and that is also the only ABI that is defined, afaik.

[dubiousjim]

Re the modules being independent of any particular port: but they'll only run on the same architecture as built for, right? I thought the target arch was specified in the .mpy format. Acknowledged that there can be varying configurations chosen for a given arch.

So if I understand, in an ideal codebase, the many header files which get recursively included by dynruntime.h would hide the elements that aren't supposed to be used in natmod builds (e.g., with a #if !MICROPY_ENABLE_DYNRUNTIME...#endif). I appreciate that would take much work. As a more-accessible fallback, there'd at least be some guidance like "don't rely on any parts of the API except what's in dynruntime.h or ..." (Are there other headers that can be relied on? Like obj.h or runtime.h?)

One of the ABI issues I noticed is that there is a thread mutex qstr_mutex conditionally included in the middle of the mp_state_vm_t struct in mpstate.h, and in my port build that's present (alternatively another mutex gil_mutex later in the struct may have been included), but in the default examples/natmod builds, MICROPY_PY_THREAD defaults to (0) so both of these fields are absent. So if a natmod accessed that struct, its later parts would be defined differently than in the port.

[jonnor]

Yes, they are architecture specific - but not (intended to be at least!) port-specific. As you mention, ports (and boards and any custom build on top of that) may differ also in their configuration - even if they are the same architecture. Most of this is about external modules being enabled-disabled, but there are a bunch of options in the "core" as well.

My point was that the natmod builds cannot know the particular configuration used by a firmware build. So we should strive to have the things there be "portable". I do not know exactly which headers can be used safely at the moment or not. Documentation of your findings here would be relevant. Especially for things which modules should be able to do.

I do no think that (most) modules would need to access mp_state_vm_t ?

[dubiousjim]

Yes, wasn't pursuing a use case where I'd access that structure. Was just trying to compile a reference for myself of the API available to natmods. And was surprised to discover divergences like this.

[jonnor]

Natmods are not very mature yet. It was only from version 1.24 that they could be used at all. In any version before, any module that used any static data would crash. So testing is much appreciated!