CBOR support?

[jeremyherbert]

It seems that micropython has great support for JSON, but there seems to be not much for CBOR. CBOR is like json but is binary encoded so it is faster, and supports typing of fields (for example, it makes a distinction between byte strings and text strings). It also is very well standardised under RFC 7049: https://tools.ietf.org/html/rfc7049

I personally have found it much more useful than json for sending data over serial links because I don’t need to constantly base64 encoded and decode binary data.

I propose adding a module which wraps the https://github.com/intel/tinycbor library, which is used in a lot of embedded frameworks (mbed, RIOT, etc) which can be added in with a #define. Is this something which is likely to be accepted as a pull request, given that it would add in a new binary library?

[stinos]

Related: #4241

[dpgeorge]

I would recommend first trying to build this as a dynamically loadable native module. There are lots of examples (and some documentation) to help get started, see the examples/natmod/ directory.

[jeremyherbert]

I think I have most of this module sorted, but I'm a bit stuck on iterating through a dictionary in a native module; I basically need to have the .items() method from python.

As far as I can tell from looking at the dict_view_print function from objdict.c, I need to expose the following extra functions via mp_fun_table: dict_items, dict_view_getiter and dict_view_it_iternext to loop through all of the items. Is this the correct way to go about this?

My current implementation (not finished) is here: https://github.com/jeremyherbert/micropython/blob/3a41f2e935fb33f070b77ad77fc3bd2488d4d922/examples/natmod/ucbor/ucbor.c

related to #5643

[dpgeorge]

I think I have most of this module sorted, but I'm a bit stuck on iterating through a dictionary in a native module; I basically need to have the .items() method from python.

The way to access an iterator using the current code is something like this:

mp_obj_t dest[2];
mp_fun_table.load_method(dict_obj, MP_QSTR_items, dest);
mp_obj_t dict_iter = mp_fun_table.call_method_n_kw(0, 0, dest);
mp_obj_iter_buf_t iter_buf;
mp_fun_table.getiter(dict_iter, &iter_buf);
mp_obj_t item;
while ((item = mp_fun_table.iternext(&iter_buf)) != MP_OBJ_NULL) {
    // use item
}

[jeremyherbert]

Thanks, got it sorted. I think the CBOR module is done now, so I would appreciate a review if possible. Would also like to hear your thoughts on how to take this forward (is there a folder in the source tree for native modules, should I keep this in a separate repo or should this be added into the extmod folder, etc)

ucbor module commit is here: jeremyherbert@999225c

changes to mp_fun_table are here: jeremyherbert@eb5c327

[AmirHmZz]

@jeremyherbert Thanks for the effort to implementing cbor in micropython.
I just wanted to know is there any further steps to merge this feature into micropython?
I know there's a lib called cbor2 in micropython-lib repository which is written in micropython. But the RAM usage seems to be significant. Having cbor in C seems more reasonable to me.

[jonnor]

Dynamic native modules can be built standalone and loaded at runtime as a .mpy file. So there is no need to merge it into the MicroPython repository, but can be done in a standalone repo. Such a project would ideally provide pre-build .mpy files that people can download directly to their devices. See https://github.com/emlearn/emlearn-micropython for one example of this.

[mattytrentini]

There's a reasonable Python-based, mip-installable (mip.install("cbor2")) option, but as @AmirHmZz points out, there may be some benefit to consider a native implementation. I think the issue is resolved but will convert this to a discussion to continue considering the native option.