Human readable version of sqlite Registry blob data

[jayich]

We are trying to use the python sqlite3 module to extract blob data from registry values that are human readable.

sqlite3 (I think) converts the blob to a Python buffer type. Any thoughts on how to convert the buffer type to a string would be great. I think that ()'s and such that are entered in the Registry are not getting converted nicely to a string. It would be great if we could get a human readable output that looks identical to what we see in the web Registry editor.

I am wondering if what we want to do is related to this PR: #84 and possibly we can't get nice human readable output at present, at least without great effort. This would also be good to know.

We are working on converting sqlite database to something human readable (yaml) that we can meaningfully git.

[maffoo]

The registry backend does store the data as binary bytes, using the same "flattened" format as is used for sending labrad data over the network. For each value in the registry It stores two things: the data as a binary blob and a string containing the type tag, which is needed to actually interpret the bytes. From these two, you can reconstruct the data, for example in pylabrad you can create a labrad.types.FlatData object from the bytes and the type tag. (You might need to convert whatever buffer type sqlite returns into a plain old byte string for things to work properly, and you also need to know that the registry stores things in big-endian byte order, represented by the string '>' in the pylabrad types code.)

If you want to convert labrad data into the text representation used by the registry editor, you can call the manager's data_to_string method (or, in the latest version of the manager, data_to_pretty_string which pretty-prints the data and so is helpful when storing more complex structured data in the registry). Be aware that we didn't intend for this textual representation to be an archival format for the data, which is why the underlying registry storage uses the flattened binary format. In practice the text format used by scalabrad has been fairly stable, but we reserve the right to change it. As an example of a way to store labrad data as text without relying on the registry editor's text format, the datavault server has routines for converting labrad data to/from data URLs that could be embedded in a yaml file, say. However, these data URLs would just be opaque strings, so you wouldn't get any useful information when looking at a git diff, say.

The pretty-printing I mentioned above has somewhat superseded #84 for our use case, because it just autoformats the registry data. We could however add a setting to the registry as in that PR to get registry data as a string. This would avoid the need of having to get the data and then call a separate manager setting to convert to a string format.

Finally, I'll just say that I'd recommend not trying to access the sqlite database directly, because it somewhat breaks the encapsulation provided by the registry server. We may change the backend data format (as proposed in #84) which would break any code that tries to access the database directly, though code that goes through the normal server interface would be fine. It's also possible to corrupt the data if you're not careful so again it'd be best to let the registry server be the only thing that touches the database, if possible.

That said, the registry does support different backend formats for storing its data. The default is to use a single sqlite database, but it can also use a "file" backend where each registry key is stored in its own file, and registry folders are stored in hierarchical folders on the file system. This also uses a binary format by default for storing the registry values themselves in the individual files. There is an option to instead store the data in individaul files in a text form compatible with what was used by the old delphi registry, but that is really just for compatibility with old registry data and I would not recommend using it. We could add another option to use individual files with a text format as used by the registry editor itself, and then you could just use git directly on the registry folder to get version control. But in that case my hesitations about using a text format for archiving data apply, so I'd suggest caution.

[jayich]

@maffoo , this is very helpful. Thank you.

Noted regarding potential changes to the scalabrad text format. Can we expected data_to_string to be fairly stable if we go this route? Thank you for the heads up about not directly accessing the sqlite database.

Regarding the very last bit of your comment: I am not in favor of adding text formatted files for registry data as the resources can be better spent elsewhere.