You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
partition_html() silently drops the text inside any element the v1 HTML parser has no class for. Every unregistered tag gets a DefaultElement, and DefaultElement.iter_text_segments() (unstructured/partition/html/parser.py) emits only the element's tail, so the element's own text and everything nested in it are lost without a warning.
For <font> this is a regression: before the parser rewrite in #3218 (0.15.0), font was listed in TEXT_TAGS (git show c27e0d00^:unstructured/documents/html.py, line 34).
Content this loses in practice:
<font>, still common in email HTML (Outlook and Gmail reply headers), so .eml/.msg HTML bodies lose text;
Inline XBRL in SEC 10-K/10-Q filings: <ix:nonFraction> wraps the numbers inside sentences, and <ix:nonNumeric>/<ix:continuation> wrap whole notes, tables included;
On the repo's own fixtures, measured as the share of word tokens from a reference text (built from each fixture's HTML, with <head>, <script>, <style> and hidden elements removed) that appear in the output:
fixture
elements
token coverage
example-docs/example-10k.html
314
66.0%: the cover-page values and all of Notes 1–14 are missing
example-docs/eml/email-no-utf8-2008-07-16.062410.eml, HTML body
2
31.5%: the replies and their From:/Sent: headers are inside <font>
Expected behavior
The text of an unrecognized element is kept, the way a browser shows an unknown element (display: inline), with any block elements inside it partitioned as usual, as if the element were a <span>. Elements whose content isn't displayed as text, such as <select>, <svg> or the hidden Inline XBRL <ix:header>, should still be skipped.
#3842 tracks this as a feature request ("custom HTML tag support"), with the open question of whether an unknown element should be treated as a block or as inline. The parser already turns block items nested in phrasing into their own elements, so an unknown element that is transparent phrasing handles both a few wrapped words and a wrapped section with headings and tables. I have a fix with tests and will open a PR that references this issue.
Describe the bug
partition_html()silently drops the text inside any element the v1 HTML parser has no class for. Every unregistered tag gets aDefaultElement, andDefaultElement.iter_text_segments()(unstructured/partition/html/parser.py) emits only the element's tail, so the element's own text and everything nested in it are lost without a warning.For
<font>this is a regression: before the parser rewrite in #3218 (0.15.0),fontwas listed inTEXT_TAGS(git show c27e0d00^:unstructured/documents/html.py, line 34).Content this loses in practice:
<font>, still common in email HTML (Outlook and Gmail reply headers), so.eml/.msgHTML bodies lose text;<ix:nonFraction>wraps the numbers inside sentences, and<ix:nonNumeric>/<ix:continuation>wrap whole notes, tables included;<hauptteil_AB>case in bug/content within custom HTML tags is skipped #3708).Everything routed through
partition_html()is affected, including.md,.epub,.rstand.org.To Reproduce
Output on
main(0ca5563):On the repo's own fixtures, measured as the share of word tokens from a reference text (built from each fixture's HTML, with
<head>,<script>,<style>and hidden elements removed) that appear in the output:example-docs/example-10k.htmlexample-docs/eml/email-no-utf8-2008-07-16.062410.eml, HTML bodyFrom:/Sent:headers are inside<font>Expected behavior
The text of an unrecognized element is kept, the way a browser shows an unknown element (
display: inline), with any block elements inside it partitioned as usual, as if the element were a<span>. Elements whose content isn't displayed as text, such as<select>,<svg>or the hidden Inline XBRL<ix:header>, should still be skipped.Environment Info
mainat 0ca5563 (0.27.8)Additional context
#3842 tracks this as a feature request ("custom HTML tag support"), with the open question of whether an unknown element should be treated as a block or as inline. The parser already turns block items nested in phrasing into their own elements, so an unknown element that is transparent phrasing handles both a few wrapped words and a wrapped section with headings and tables. I have a fix with tests and will open a PR that references this issue.