Repository navigation
Make extraction faster without changing output - #24
Merged
Merged
Conversation
Boilerplate#walk and Pruner#remove_secondary built a NodeSet of each element's children and iterated it in Ruby. Following first_element_child/next_element visits the same elements in the same order without the NodeSet. remove_secondary reads the next sibling before removing, and a spec covers removals next to and inside kept elements. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A container with no text containers had every span, div and list summed before the total was compared with zero; the check now returns at the first child holding text and only searches the subtree when no child decides. The embeds and list items are queried once, the paragraphs are only counted when there are more than 100 list items or any input, and short_with_links? tests the length before the density. Specs cover each rule. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Renderer, Inline, Table and CodeBlock listed children through Node#children, which builds a NodeSet and wraps each node again when it is read by index. Dom.children follows the sibling pointers and returns the same nodes as an Array. Rendering does not change the DOM, so the list cannot go stale. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
remove_stray_lines and remove_boilerplate_text squished the text of every candidate block only to drop those longer than 120 or 200 characters. Dom.min_text_length counts the characters squish keeps without building the string, so long blocks are skipped first. Stray lines now check for an enclosing table only once their text matches. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The redirector pattern is anchored once, so most hosts fail at the first character instead of in every branch at every position. A URL is only percent-encoded when it holds a character that needs it, and only a URL with a fragment is parsed again to see whether it points into the page. Resolving the 2,164 links of a long Wikipedia article on a fresh resolver takes 9.4 ms instead of 14.8 ms. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Only a heading, an element inside one or the section one opens can carry a heading's id, so marking reads the elements around each heading instead of every element with an id on the page. Marking takes 0.66 ms instead of 5.95 ms on a long Wikipedia article, and 0.17 ms instead of 0.36 ms per page on WCXB. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Nokolexbor builds a NodeSet for element_children and wraps every node again when it is read. Dom.element_children walks first_element_child and next_element instead, as Dom.children does for all children. It is used where rendering, paragraph detection, content location and review removal read element children. On a long Wikipedia article, finding which inline elements hold blocks takes 4.8 ms instead of 6.7 ms when the whole page is rendered. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Most rendered paragraphs are one line with no doubled space, no separator and no line-start syntax, yet each went through a split, two tidying gsubs, a split into lines and three escaping gsubs. Each pass now runs only when the text holds what it looks for. Output is unchanged on every corpus page. Text paragraphs take 0.66 instead of 1.26 ms per WCXB page in full mode, and 3.1 instead of 7.0 ms on the Wikipedia page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Lexbor's XPath engine finds the junk tags in about half the time its selector engine takes for the same comma list. The set of elements is the same, and removing them in a different order changes nothing. Wikipedia Coffee preprocessor: 14.7 -> 12.8 ms (main), 14.3 -> 12.3 ms (full). Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Every piece of inline text was matched against the email pattern, which needs an "@" to match. Checking for the "@" first skips the regex on nearly every piece. Wikipedia Coffee, full page, Inline#append: 5.6 -> 4.0 ms. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate matched every class signature against the HEADER and MENU patterns. Both need a word, "header", "masthead" or "menu", that most signatures lack, so checking the word first skips the patterns. A spec spells out every name each pattern matches and checks it holds the word. Wikipedia Coffee boilerplate stage: 39.8 -> 38.0 ms. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The scorer, the content locator and the pruner each ask whether the same names are secondary regions, so the answer is now kept with the signature cache for the extraction and dropped with it. Wikipedia Coffee, Patterns.secondary?: 4.5 -> 2.0 ms. WCXB main, every 4th page: 16.98 -> 16.78 ms/page. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Candidate scoring and conditional cleaning weigh the same class lists again and again: on WCXB, a page asks for about 18,000 distinct weights in 128,000 calls. The weights are now kept with the signature cache for the extraction and dropped with it. WCXB main, every 4th page, inclusive ms/page: words_weight 0.34 -> 0.17, cleaning_weight 0.41 -> 0.28, class_weight 0.24 -> 0.18. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate removes noise widgets by their class list after stripping WordPress and WooCommerce terms from it. A page repeats its class lists, so the verdict for each signature is now kept for the pass instead of rebuilt for every element. WCXB main, every 4th page, inclusive ms/page: Boilerplate#furniture_class? 0.93 -> 0.78, Boilerplate#call 3.73 -> 3.51. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stats kept characters and link characters in two hashes, so text_chars and link_density read the element's key and the hashes twice. One entry per key now holds both counts, and the key is frozen before it is stored, so Hash no longer copies it. WCXB main, every 4th page, inclusive ms/page: Stats#initialize 1.74 -> 1.52, Stats#text_chars 0.48 -> 0.18, Stats#chars 0.24 -> 0.04. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
UrlResolver.normalize searched every href for its first and last visible characters. Most values start and end with one, which two bytes show; only then are line breaks and tabs dropped as before. Other values take the search, and invalid UTF-8 still raises as it did. Wikipedia Coffee, full page, inclusive with tracing: normalize 3.9 -> 2.0 ms. Output is identical on every corpus page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A link that holds only text has no image to drop, so thumbnails returns before searching it. Most links on a page are text: 798 of 943 on a long Wikipedia article, and 17,066 of 25,737 over WCXB. Finding thumbnails takes 0.99 ms instead of 3.08 ms on the Wikipedia article, and the whole pass 10.0 ms instead of 12.4 ms. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The Pruner judges every link, span, div and paragraph before removing any, so the order they are found in does not matter, and an XPath union finds a list of tags faster than the CSS selector list did. Removing calls to action takes 3.3 ms instead of 5.8 ms on a long Wikipedia article. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
button? built and downcased a string of the class and id for every link, span, div and paragraph. Matching each attribute case-insensitively answers the same, since no Unicode letter folds to the ASCII letters of btn, button or cta, and allocates nothing. Checking for buttons takes 1.9 ms instead of 2.7 ms on a long Wikipedia article (instrumented). Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
paragraph_keys only collects the breakers' ancestors and the candidate paragraphs into sets, so the order they are found in does not matter, and an XPath union finds a list of tag names faster than a CSS selector list. Finding paragraphs takes 3.2 ms instead of 4.4 ms on a long Wikipedia article, and 0.58 ms instead of 0.63 ms per page on WCXB. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The redirector pattern is anchored once so most hosts fail at the first character, but its unanchored safelinks branch made the engine try every position of every link anyway. Matched on its own at the end, it answers the same. Checking for redirectors takes 0.75 ms instead of 1.29 ms on a long Wikipedia article (instrumented). Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
tr reads a non-ASCII string as characters, which on large subtrees cost three times what a byte-level replacement does. Valid UTF-8 now has its non-ASCII spaces replaced as byte sequences and its ASCII spaces with tr on the bytes; other strings take tr as before. spaces_to_blanks over every 4th WCXB page: 1.08 -> 0.69 ms/page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stats walks every node of the page twice per extraction. It now reads each child's type once and compares it with the node type constants, instead of calling text?, cdata? and element?, which read it again each. A link's weight reads the href first, since most elements have none. On a long Wikipedia article, counting the page takes 6.9 ms instead of 7.2 ms. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Resolving a link parsed it with URI, joined it to the page's address and cleaned its query, even when it had no query, nothing to encode and no dot segment. A plain web URL now comes back as it is, and a plain path from the site's root after the page's origin; Amazon's links, and a page that is not on the web, still go through URI. Over the links of a corpus of web pages, both ways give the same URLs, and 82% are plain. On a long Wikipedia article, resolving its links takes 3.0 ms instead of 10.1 ms when the whole page is rendered. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Few pages carry ad-targeting attributes, yet the walk asked every element for each of six names. One selector query over the body now tells whether any exist, and the per-element check runs only then. Wikipedia Coffee: advertising? 3.1 -> 1.6 ms. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A cached signature was looked up by its class list and id joined into one string, which every call built again. Two lookups, by class list and then by id, find it without building anything. Signature lookup on Wikipedia Coffee: 309 -> 246 ns per call. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Each inline node and list child was asked text?, cdata? and element? in turn, a method call each that reads the type again. The renderer now reads the type once and compares it with the constants, as the character count does. Wikipedia Coffee, full page: 33,000 predicate calls fewer. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Conditional cleaning asked every container whether it held a player, an embed, an input, an image or a list item. Most roots hold none of some of these, and removing containers never adds any, so the root is asked once and its containers skip the question when the answer is no. WCXB remove_suspicious_containers: 1.51 -> 1.38 ms/page. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Lexbor matches a selector list one selector at a time, walking the tree once for each of the 27 hidden-candidate selectors and then sorting the results. Wrapped in :is(), the list is matched in one walk and returns the same elements in the same document order. WCXB remove_hidden: 0.62 -> 0.50 ms/page; Wikipedia Coffee query 2.65 -> 1.40 ms. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Most inline runs with a line break have no space or tab beside it, yet each one went through a gsub that rewrote every break. That gsub now runs only when a blank touches a break, as two anchored passes, and doubled spaces are squeezed instead of matched by a pattern. Output is unchanged on every corpus page. tidy takes 0.92 instead of 1.14 ms on the Wikipedia page in full mode. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Listing what is under every table walks a nested or later table's elements once for each table above or before it, so the time grew with the square of the number of tables: 2.2 seconds for 800 small tables. The outermost tables hold every table element, and listing the elements of each takes time in proportion to the page. An article of 1,000 sections, each a paragraph and a table, takes 1,316 ms instead of 5,165 ms (1,684 ms before the table check moved). Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A heading's parent was read once per heading in it. When that parent is inside an unclosed heading, its label is the text of the whole section, so a page with a mistyped </h1> and hundreds of sections took seconds where the elements were read once before. Each parent is now read from the first heading in it, and not at all when it is read as a heading or as a heading's child already; a heading in a heading is read as a heading only. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Every button read the whole text of each element around it, to see whether the element held nothing else, and listed every link of its card. An article of paragraphs each followed by a button read the article's text once per button: 3.3 s for 2,000 buttons and about two minutes for 8,000. Buttons are judged before any is removed, so an element's text length and a card's links are now read once and kept for the others; the text is compared only when the lengths match. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Reading the elements around a heading costs about as much as checking eight elements for an id. On a page of thousands of empty headings and one id, marking took 9 ms where checking every element took 1 ms, so the page was slower than before with a short max_length. Where there is a heading for fewer than eight elements, every element with an id is read instead, as before; the two ways mark the same elements. The list of elements is made once and also numbered. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A parent was read from its first heading, found by stepping back over the parent's other children for every heading in it. A section of headings each followed by a few elements was slower than reading every element with an id. Parents read are now listed by id, so each is read from the first heading met in it. Past eight elements sharing an id, the step back is used again, so duplicate ids stay cheap. The marking specs now hold the filler they meant to, so they run the path around headings. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Parents were listed by their id, which was read and hashed for every heading, so a long id cost its length once per heading. Past eight parents sharing an id, every heading stepped back over its siblings again. A heading that shares the last heading's parent is not that parent's first, so only a heading with a new parent reads its id and steps back to the heading before it. Siblings are stepped over once at most, and the same elements are marked. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A parent has a label only when the heading opens its section, after nothing but empty elements. Each new parent stepped back from its heading to the one before it, so headings that took turns between a section and a nested element stepped back over every gap again. The check now also steps on from the section's first element and stops at the first one with content, a child or a heading, so a section that opens with text or another heading is told at once. Stepping back still reads each element once over all headings. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a button is a card's only link kept an Array of every link node of each card. Nested cards each held the links they share, so peak memory doubled on pages of nested boxes that each open with a button. The card's links are now kept as the node set the query returns, which holds no Ruby nodes, together with the first nine distinct keys of the links that lead elsewhere. A button that lacks one of those keys is not the card's only link, so the set is read again only for a button that holds them all. Output is unchanged. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A card's first nine distinct link keys were read in full for the first button in it, so a card whose first link leads elsewhere still read every long same-page link after it. The keys are now read in order only as far as some button has needed, and a button that holds them all goes on from where reading stopped. No link is read twice, and the scan stops at the first link that leads elsewhere than the button, as before. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Headings that take turns between a section and an element nested in it made every heading tell again whether it opens the section, stepping over the empty elements before it each time. The checks now share a budget of one step per heading plus a sixteenth of the page's elements. They run before anything is read, so when the budget runs out every element with an id is read instead, which marks the same elements. No WCXB page uses more than a twelfth of the budget. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Blocks were rendered from a list of all of an element's children, built before the first was read, so a budget spent early still listed every child of a long element. The children are now read through the sibling pointers as rendering goes, and stop when the budget does. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a heading opens its parent's section stepped on from the parent's first element. Finding that element, and the next one, passes every text node before it, and a section whose opening text was cut up by removed scripts or comments was passed again for every heading: quadratic in the page. Step over child nodes instead, text included, so each heading takes one step per node until the walk stops. The answer is the same: elements without content are still stepped over, and one holding an element, or a heading, still ends the walk. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Listing the sections headings open looked up each parent's id, and marking looked it up again. A parent whose headings take turns with headings nested in it is listed once per turn, so a parent with many attributes had them read twice per heading. List only the sections a heading opens, telling each without its id, and look the id up once, when the section is marked. Marked elements are the same: the id is still read after the heading is marked, where a misnested element's copies share their attributes. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a heading opens its parent's section read the text of every element before it, to see whether it is empty. Since every listed parent is told, with an id or without, a section without an id had its opening text read, which marking it never needed. Stop stepping back only at a heading or an element that holds one, without reading text, and leave whether the elements before the heading are empty to the label, read only for a section with an id. That label is the one every element with an id had read before, so the same elements are marked. Stepping back stops at the heading before, so it passes each element once over all headings, and only steps on are taken from the budget. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The parser repeats an element left open, and the copies share one list of attributes, but a mark added through a copy that is not the first of its element is lost when the first one gains an attribute. Marking in the order of the headings could mark a later copy first, so the id was then read as a name: "social-media" as a share bar, and every paragraph holding a copy was removed. Find the anchors around headings first and mark them together. Where two of them are copies of one element, with the same id, read every element with an id instead, which marks in document order. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stepping back from a heading took no step from the budget, so where each section opens with text and then many empty elements, every one of them was passed, where the budget would have read every element with an id instead, which is faster there. Take every step from the budget again. The text-free check on the elements before the heading stays. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When two elements the parser repeats shared an id, the anchors around headings were read, then read again by every element with an id. A page of sections whose headings hold the same anchor read each label twice. List the elements with an id around the headings first, and read their labels only when no two copies share an id, so each element is read once either way. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether two listed elements were copies of one misnested element read the id of every element the parser repeats, including those whose label no id could match, and hashed it. A long id shared by many copies was copied once per copy. A later copy shares its first copy's attribute list, which that first copy owns, so asking an id attribute for its element tells a later copy without reading the id. Elements apart, even with the same id, own their attributes and no longer fall back to reading every element with an id. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An attribute of a misnested element's copies names as its element the first copy it is asked for through, not always the first copy in the document. The check still holds, as two listed copies cannot both be named, and a copy listed alone is marked as in document order. A spec asks the copies for their id in reverse first. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
On Ruby 3.4 and 4.0 the first measured block allocates one object more, and the garbage collector keeps short-lived nodes found by a query alive past GC.start, so two specs failed there by a margin of one or two. The signature spec now takes the fewest allocations of three runs, and the card spec counts the nodes the pruner holds instead of the live ones. Both still fail on the lib before the change each one covers. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The copies Lexbor leaves of a misnested formatting element share one attribute list, and an attribute written through one copy after keys are set (a forum's thread mark, a replaced member panel) can drop the key of another. Stats then had no counts for that copy and returned nil, so scoring a forum page raised NoMethodError in text_chars or link_density, or TypeError when Boilerplate summed the characters of a run of cards. An element without a key is now counted each time it is asked about, and nothing is stored for it. Pages that raised now return a result, and every other result is unchanged. Looking up the link around an element walks its parents instead of building the list of its ancestors. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A keyless element was counted with the link weight of the links around it, but not its own, so a keyless <a href> read as prose. On a forum page whose reply link runs into the reply, that passed the reply alone as the article and dropped the question above it. The element's own href now counts, as it does in the walk from the root. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An element that lost its key was counted again on every lookup. Boilerplate asks for an element's parent once per phrasing child, so a bold copy holding N spans was walked N times, and a 150 KB forum page that used to raise after 12 ms took 15 s. Its counts are now held, by node among the elements from the same source position, until Stats removes nodes or a thread is marked, the two writes that can change them. The same page takes 65 ms, and every result is the same as before. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Source positions restart in every fragment the page parses, such as each <noscript> fallback, so unrelated elements without a key can share one position. Searching every count held there made a page of many such fragments quadratic. Keep only the 16 most recently used counts for each position. A lookup then compares at most 16 nodes, and an element asked about once per child stays held, so the repeated-parent case stays linear. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Holding only the 16 most recently used counts per source position let elements from parsed fragments at the same position push out an element that is asked about once per child. Each push meant counting the whole element again, so such pages were quadratic again. Also hold each count in a list for its place: the key of the nearest ancestor with one, how many elements without a key lie between, and the source position. Nothing inside an element shares its place, so asking about its children cannot push it out there. A lookup still searches at most two lists of 16. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Sibling fragments, each parsed from its own <noscript>, can put any number of elements without a key at one place: the same nearest keyed ancestor, depth and source position. So can copies of a misnested ancestor, which share its key. With more than HELD of them asked about in turn, as the class-group pass does, each lookup pushed out the next one wanted, and every element was counted again in full each time. A place now lists its first HELD elements and pushes none out. Once another shares it, the place holds each by its path, which no other element in the tree has, and still finds it by node. Each element is counted once until #remove or #forget_keyless, and a lookup still compares at most HELD nodes in each list it reads. Only pages with more than HELD elements at one place ever ask for a path. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When more than HELD elements without a key share a place, the ones beyond the place's list were each found by their path. Reading a path walks every earlier sibling at every level up to the root, so with thousands of sibling fragments that cost more than it saved. They are now listed first by their spot: the key of the nearest element before or above them that has one, and the steps back or up that reach it, which lead to the element alone from each element with that key. Only elements sharing a full spot list, or with no key within SPOT_STEPS, read their path. A lookup compares at most HELD nodes in each of up to four lists. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An element without a key at a full place list was found by a nearby key, and then by its path. Node#path counts the siblings before the element at every level, so many such elements under one wide parent were found in quadratic time, and a page of 800 foster-parented fragments took 15 to 19 s, against 6 to 7 s before the path list. A place now lists every element asked about there and pushes none out. A list of up to 32 keeps the most recently used last, as before. A longer one keeps its order: a search looks beside the last element found first, since elements are mostly asked about in document order, and the entries are also kept in NodeSets of runs of 32 and of 1,024, which compare nodes in C. So a search compares one run of 32 in Ruby and walks nothing in the tree. The spot and path lists go. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes extraction faster without changing any output. Each public result (
Nokodown.extract(...).to_handto_markdown) is meant to stay byte-identical tomain(3dc2249) for every input and option set, except on pages that raised onmain(see the fix below), which now return a result. This includesmax_length,images:/links:,main_content: false, malformed or deeply nested HTML, and other encodings.Speed
CPU ms, best runs,
main→ this branch:mainbenchmark/speed.rbon WCXB (2,008 pages), wall ms on a loaded machine:Some page shapes that were quadratic on
mainare now linear:What changed
Keeping output the same
main's queries or queries one selector at a time.Verification
bundle exec rspecruns 520 examples with 0 failures and 1 optional pending.main); every run that did not raise onmaingives the same result.main: 0 differences on several hundred thousand generated pages, other than pages that raised onmain. These cover malformed and unclosed markup, nested headings, misnested formatting elements with ids, tables, buttons and cards, encodings and nesting depths up to 30,000. They also cover forced dense and sparse anchor-marking paths and threaded, nested and repeated calls.main.Fix: pages that raised on misnested formatting
Some malformed pages raised
NoMethodErrorinStats#link_density/#text_chars, orTypeErrorinBoilerplate#card_run?, onmaintoo.<b>,<i>,<a>, …), and the copies share one attribute list, but each copy keeps its own pointer to the last attribute. Writing an attribute through a copy with a stale pointer drops the attributes added through the others, including the element's key. The thread and post-attribution marks write after keys are set, soStatsthen had no counts for that element.NodeSets of runs of 32 and 1,024, so finding an element never walks the tree.