XML to MarkDown conversion

[akutuzov]

Efficient code to inject into the general HPLT pipeline

[jindrahelcl]

the script is now merged

[jindrahelcl]

see statistics on reasons to ignore a given line in the data. this was computed on /scratch/project_465001890/zaragoza/monotextor-processing/collections_merged/ces_Latn/batch_1.jsonl.zst

[jindrahelcl]

code introduced in #10

The conversion script is at https://github.com/hplt-project/HPLT-textpipes/tree/main/tools/xml2md

[oepen]

from glancing over some outputs, it appears as if the escaping revisions yesterday escape all periods? why is that necessary, and what about the risk of breaking in-line links which are just an URL?

[jindrahelcl]

hmm, I agree that this is probably too aggressive. I was using the list of escapable characters from here which does include full stop, exclamation mark, and some other more obvious choices.

Inline URLs do get broken..

[jindrahelcl]

Better escaping pull request here #12

[oepen]

escaping tends to raise tricky questions :-). my markdown expertise is still growing, but for some of the "escapable" characters, i would think they only are special in certain contexts, e.g. # (for headings) at the start of a line, | (cell separator) inside a table, + or - (list item) at the start of a line, ! (image) and ( and ) (url) only in connection with a link, i.e. [ and ]. for . (dot), i cannot even say where it is used as a special character? other special characters will likely always or almost always require escaping, e.g. \ (for a literal backslash) and * or _ (in-line italics or bold).

given these considerations, there are probably trade-offs between "playing it safe" (lots of escaping, even in contexts that do not require it) vs. escaping much like a human would do. the latter could be said to yield more "readable" markdown.

[jindrahelcl]

At this point, backslashes, backticks, * and _ are always escaped, #, +, and - are only escaped when on beginning of a line, pipes only in tables, and square brackets inside link texts. Dot is escaped so we don't accidentally create an ordered list (when a number is followed by a period at the beginning of a line). The script also escapes what would render as a horizontal line, quote, and definitions (lines beginning with a colon).

[oepen]

some findings from a joint markdown excursion.

this looks like a good standard, assuming basic plus extended syntax: https://www.markdownguide.org/cheat-sheet/

• \, `: always escape
• #, +, -, >, :: escape at start of line (can be indented with whitespace)
• ~: escape when in pairs and not followed by whitespace
• <: escape when paired with closing > on same line, and non-whitespace between them
• .: escape when following a line-initial number

maybe try a linter like this one https://pymarkdown.readthedocs.io/en/latest/

[oepen]

we ran out of time yesterday to look more closely at *, _, **, etc. for italics and bold. it appears they need to be immediately followed or preceded, respectively, by non-whitespace to act as markup (at least on github) and can span multiple lines, but not paragraph boundaries. consider:

foo *bar
baz*

foo * bar *

foo *bar

baz*

which renders as:

foo bar
baz

foo * bar *

foo *bar

baz*

in our trafilatura outputs, whitespace has been normalized, and accidental linebreaks (breaking the HTML source across multiple lines) been converted to just whitespace. the \n in our text and (most likely also) xml strings are segment boundaries, i.e. they were inserted according to paragraph-like breaks in the original document, typically indicated by HTML elements like <h>, <p>, <li>, etc. thus for the purpose of checking pairs of font markers like * … *, as suggested above, i imagine it will be sufficient to break the XML string into segments according to the \n boundaries, and look for matching pairs within each segment. quite likely the same applies to some of the other paired markers, e.g. < … > and ~~ … ~~.

if we want inclusion of markdown as a distinguishing property of the HPLT 3.1 release, i fear we need to try and get escaping to a level that is (by and large) syntactically well-formed and plausible to a human reader, e.g. avoiding over-escaping as good as we can.