Log sources and history of derivations and lexemes

[vidraj]

The old Perl API logged derivation history and remembered which module and which annotation file was responsible for each derivational link. This is very useful when debugging – if you see an error in the database, you can quickly trace it to its source by looking at the history.

It is also theoretically useful for identifying homonyms – we can use the derivation history to find lexemes that have multilple legitimate parents, without complicating the API by relaxing the 'single parent' restriction. These can be dual paths or homonyms.

The Python API lacks this capability so far, but it should be added.

Ideally, we would track the following for each link:

• the linked lexeme
• the module that created the link
• the file where the link was found (if applicable, some modules don't need/use annotations)
• the line number where the link was found (ditto)
• maybe also the exact line / paragraph for easy grepping

and remember the whole history of these informations.

Lexeme and module information can be obtained automatically, because the pipeline manager knows the linked lexeme and which module is currently running. The file and line information probably has to be provided by the modules.

The Perl API also remembered lexeme creators in the same way. Lexemes can come from MorfFlex, extra annotations or specific modules and we want to distinguish these sources.

[vidraj]

History of relations is tracked as of commit 48eeb38. All additions are tracked, as well as deletions that use the new convenience methods in Lexicon: remove_relation(), remove_all_parent_relations(), remove_all_child_relations() and remove_all_relations().

Adding and removing lexemes is not tracked yet. It's not clear how to track removing lexemes, and adding them is hopefully rare enough that tracking is not necessary. Having lexeme provenience from the start would be nicer to have, as the start of the pipeline sources data from multiple origins, but also more difficult to implement.

It's not impossible, though, since we already have a direct line from the Perl API to the new Python API to import tag masks. It would be possible to also import the history from the Perl API, including lexeme history.

The correct place to do that would be version 2.0, reading from /tools/data-api/perl-derivmorpho/derinet-1-3b.tsv.