Extending the API exposed to dynamic native modules?

[dubiousjim]

Hi I'm interested in understanding better the way that functions and objects are exposed to dynamic native modules (.mpy files) in py/nativeglue.[ch], together with tools/mpy_ld.py. Especially with an eye to what is the correct way to extend these with local additions.

The doc file https://docs.micropython.org/en/latest/develop/natmod.html says:

Linker limitation: the native module is not linked against the symbol table of the full MicroPython firmware. Rather, it is linked against an explicit table of exported symbols found in mp_fun_table (in py/nativeglue.h), that is fixed at firmware build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system function, for example.

New symbols can be added to the end of the table and the firmware rebuilt. The symbols also need to be added to tools/mpy_ld.py’s fun_table dict in the same location. This allows mpy_ld.py to be able to pick the new symbols up and provide relocations for them when the .mpy is imported. Finally, if the symbol is a function, a macro or stub should be added to py/dynruntime.h to make it easy to call the function.

As of MicroPython 1.25.0,
The mp_fun_table struct defined in py/nativeglue.[ch] has 87 fields. The first 50 of these, whose last item is setjmp_, let's call group A. The next 23 fields, starting with item memset_ and whose last item is plat_print, let's call group B. The final group, starting with item type_type and ending with stream_write_obj, let's call group C. These have the following distinguishing properties. The group A items each have a corresponding enum entry in mp_fun_kind_t in nativeglue.h. I'm guessing these are handled something like (or perhaps exactly as) virtual machine opcodes. The group B and C items have no corresponding enum. However, the group C fields are specifically detected by the link_objects function in mpy_ld.py, via its fun_table dict comprehension. I don't yet understand why that's done.

Suppose we want to locally add additional items to be exposed to dynamic native modules, as I do in #17197. It would be possible to put these before group B, as an extension of group A, if one also adjusted where MicroPython thought groups C and B began. Similarly, it would also be possible to put them before group C. I'm expecting though that it would be a bad idea to do either of these things, because it would change the ABI and might break some things. (Can someone say what specifically would break? .pyc files? .mpy files compiled by toolchains without these mods? something else?)

So I'm expecting that the right place to add things would be after group C. Let's call any such additions group D. Let's subdivide that into group DC, which comes first and is an extension of group C. So any such additions would be added as fields to the mp_fun_table struct in nativeglue.h, and an object or function would be added to the initializer for that struct in nativeglue.c, and also some entry would be added to the end of the dict comprehension fun_table in mpy_ld.py. (What specifically should that final entry be? The field name from nativeglue.h? A C reference to the field value from nativeglue.c? Some other name that your dynamic native module uses? I think it's the second of these.)

The second subdivision would be group DB. This is essentially an extension of group B, but it comes after groups C and DC, and gets no modifications to mpy_ld.py. It involves modifying nativeglue.[ch] in the same way group DC does.

What I most want to understand is under what circumstances I'd want to put a local addition to what's exposed to dynamic native modules in group DC, and when in group DB.