Skip to content

Make extraction faster without changing output - #24

Merged
jasir-serpapi merged 87 commits into
mainfrom
jasir/faster-extraction
Nov 16, 2026
Merged

jasir-serpapi merged 87 commits into
mainfrom
jasir/faster-extraction

Conversation

@jasir-serpapi

@jasir-serpapi jasir-serpapi commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Makes extraction faster without changing any output. Each public result (Nokodown.extract(...).to_h and to_markdown) is meant to stay byte-identical to main (3dc2249) for every input and option set, except on pages that raised on main (see the fix below), which now return a result. This includes max_length, images:/links:, main_content: false, malformed or deeply nested HTML, and other encodings.

Speed

CPU ms, best runs, main → this branch:

Page Mode main Branch
Wikipedia, Coffee main content 205 136
Wikipedia, Coffee full page 81 49
Large article (20,000 paragraphs, 5.4 MB) main content 892 678
Large article full page 128 89
books.toscrape.com main content 1.9 1.5
W3Schools video page main content 15.2 12.2

benchmark/speed.rb on WCXB (2,008 pages), wall ms on a loaded machine:

Mode p50 p99
Main content 12.8 → 9.9 108.9 → 72.9
Full page 8.1 → 5.4 52.6 → 33.1

Some page shapes that were quadratic on main are now linear:

  • 8,000 paragraphs, each followed by a button link: 116 s → 0.7 s.
  • A 2,000-entry table of contents: 8.2 s → 0.3 s.

What changed

  • Fewer passes over the page: XPath unions replace repeated CSS queries, and sibling and element pointers replace child-list scans.
  • Memoised work per extraction: class-list weights, signatures, furniture-name checks and each element's counts.
  • Cheaper text handling: byte-level space replacement, squishing skipped for text that is already squished or too long to match, and URLs with nothing to resolve or encode returned as they are.
  • Bounded heading-anchor marking: only the elements around headings are read, under a step budget. Where headings are dense, or telling which sections they open takes too long, marking falls back to reading every element with an id.
  • Deep markup: junk elements and links more than 10,000 levels down are still found.

Keeping output the same

  • Selector lists: Lexbor renumbers the document when a CSS selector list matches. Where the order of that numbering affects later steps, the branch keeps main's queries or queries one selector at a time.
  • Misnested elements: the parser repeats a misnested element, and the copies share one attribute list. An anchor mark written through one copy can be lost when another copy gains an attribute. Copies are therefore marked in document order, and the marking falls back to the full pass whenever two of them are listed.
  • Spec for each fix: every fix found in review has a spec that fails without the fix.

Verification

  • Suite: bundle exec rspec runs 520 examples with 0 failures and 1 optional pending.
  • WCXB: all 2,008 pages give identical output in both modes. F1 is unchanged: ScrapingHub 0.947, Mozilla 0.949, WCXB dev 0.875, WCXB test 0.900.
  • Saved corpus results: 6,387 results are identical.
  • Crash fix: 0 of 40,016 generated runs raise (185 did on main); every run that did not raise on main gives the same result.
  • Differential fuzzing against main: 0 differences on several hundred thousand generated pages, other than pages that raised on main. These cover malformed and unclosed markup, nested headings, misnested formatting elements with ids, tables, buttons and cards, encodings and nesting depths up to 30,000. They also cover forced dense and sparse anchor-marking paths and threaded, nested and repeated calls.
  • Timing: each slow page shape found during review is no slower than main.

Fix: pages that raised on misnested formatting

Some malformed pages raised NoMethodError in Stats#link_density/#text_chars, or TypeError in Boilerplate#card_run?, on main too.

  • Cause: the parser repeats a misnested formatting element (<b>, <i>, <a>, …), and the copies share one attribute list, but each copy keeps its own pointer to the last attribute. Writing an attribute through a copy with a stale pointer drops the attributes added through the others, including the element's key. The thread and post-attribution marks write after keys are set, so Stats then had no counts for that element.
  • Fix: an element without a key is counted on demand, the same way the page walk counts it, link weight included, and nothing is stored under a key for it. Keys are not restored, because restoring them changed results on pages that did not raise.
  • Cost: a count is held until the page changes. It is found by source position (the 16 most recently used), then by place (nearest keyed ancestor, depth, position). A place lists every element asked about there. A long list is searched in C through NodeSets of runs of 32 and 1,024, so finding an element never walks the tree.
  • Scope: only pages that raised before give a different result: they now return one. Every other result is unchanged.
  • Specs: pages that raised now return the expected result; a keyless link keeps its weight; repeated lookups count each element once and stay bounded in compares and time on wide and fragment-heavy pages.

jasir-serpapi and others added 30 commits November 12, 2026 17:44
Boilerplate#walk and Pruner#remove_secondary built a NodeSet of each
element's children and iterated it in Ruby. Following
first_element_child/next_element visits the same elements in the same
order without the NodeSet. remove_secondary reads the next sibling
before removing, and a spec covers removals next to and inside kept
elements.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A container with no text containers had every span, div and list
summed before the total was compared with zero; the check now returns
at the first child holding text and only searches the subtree when no
child decides. The embeds and list items are queried once, the
paragraphs are only counted when there are more than 100 list items or
any input, and short_with_links? tests the length before the density.
Specs cover each rule.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Renderer, Inline, Table and CodeBlock listed children through
Node#children, which builds a NodeSet and wraps each node again when it
is read by index. Dom.children follows the sibling pointers and returns
the same nodes as an Array. Rendering does not change the DOM, so the
list cannot go stale. Output is identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
remove_stray_lines and remove_boilerplate_text squished the text of
every candidate block only to drop those longer than 120 or 200
characters. Dom.min_text_length counts the characters squish keeps
without building the string, so long blocks are skipped first. Stray
lines now check for an enclosing table only once their text matches.
Output is identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The redirector pattern is anchored once, so most hosts fail at the first
character instead of in every branch at every position. A URL is only
percent-encoded when it holds a character that needs it, and only a URL
with a fragment is parsed again to see whether it points into the page.

Resolving the 2,164 links of a long Wikipedia article on a fresh
resolver takes 9.4 ms instead of 14.8 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Only a heading, an element inside one or the section one opens can carry
a heading's id, so marking reads the elements around each heading instead
of every element with an id on the page.

Marking takes 0.66 ms instead of 5.95 ms on a long Wikipedia article,
and 0.17 ms instead of 0.36 ms per page on WCXB.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Nokolexbor builds a NodeSet for element_children and wraps every node
again when it is read. Dom.element_children walks first_element_child and
next_element instead, as Dom.children does for all children. It is used
where rendering, paragraph detection, content location and review
removal read element children.

On a long Wikipedia article, finding which inline elements hold blocks
takes 4.8 ms instead of 6.7 ms when the whole page is rendered.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Most rendered paragraphs are one line with no doubled space, no
separator and no line-start syntax, yet each went through a split, two
tidying gsubs, a split into lines and three escaping gsubs. Each pass
now runs only when the text holds what it looks for.

Output is unchanged on every corpus page. Text paragraphs take 0.66
instead of 1.26 ms per WCXB page in full mode, and 3.1 instead of 7.0
ms on the Wikipedia page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Lexbor's XPath engine finds the junk tags in about half the time its
selector engine takes for the same comma list. The set of elements is
the same, and removing them in a different order changes nothing.

Wikipedia Coffee preprocessor: 14.7 -> 12.8 ms (main), 14.3 -> 12.3 ms
(full). Output is identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Every piece of inline text was matched against the email pattern, which
needs an "@" to match. Checking for the "@" first skips the regex on
nearly every piece.

Wikipedia Coffee, full page, Inline#append: 5.6 -> 4.0 ms. Output is
identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate matched every class signature against the HEADER and MENU
patterns. Both need a word, "header", "masthead" or "menu", that most
signatures lack, so checking the word first skips the patterns. A spec
spells out every name each pattern matches and checks it holds the word.

Wikipedia Coffee boilerplate stage: 39.8 -> 38.0 ms. Output is identical
on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The scorer, the content locator and the pruner each ask whether the same
names are secondary regions, so the answer is now kept with the
signature cache for the extraction and dropped with it.

Wikipedia Coffee, Patterns.secondary?: 4.5 -> 2.0 ms. WCXB main, every
4th page: 16.98 -> 16.78 ms/page. Output is identical on every corpus
page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Candidate scoring and conditional cleaning weigh the same class lists
again and again: on WCXB, a page asks for about 18,000 distinct weights
in 128,000 calls. The weights are now kept with the signature cache for
the extraction and dropped with it.

WCXB main, every 4th page, inclusive ms/page: words_weight 0.34 -> 0.17,
cleaning_weight 0.41 -> 0.28, class_weight 0.24 -> 0.18. Output is
identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate removes noise widgets by their class list after stripping
WordPress and WooCommerce terms from it. A page repeats its class lists,
so the verdict for each signature is now kept for the pass instead of
rebuilt for every element.

WCXB main, every 4th page, inclusive ms/page: Boilerplate#furniture_class?
0.93 -> 0.78, Boilerplate#call 3.73 -> 3.51. Output is identical on every
corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stats kept characters and link characters in two hashes, so text_chars
and link_density read the element's key and the hashes twice. One entry
per key now holds both counts, and the key is frozen before it is stored,
so Hash no longer copies it.

WCXB main, every 4th page, inclusive ms/page: Stats#initialize
1.74 -> 1.52, Stats#text_chars 0.48 -> 0.18, Stats#chars 0.24 -> 0.04.
Output is identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
UrlResolver.normalize searched every href for its first and last visible
characters. Most values start and end with one, which two bytes show;
only then are line breaks and tabs dropped as before. Other values take
the search, and invalid UTF-8 still raises as it did.

Wikipedia Coffee, full page, inclusive with tracing: normalize
3.9 -> 2.0 ms. Output is identical on every corpus page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A link that holds only text has no image to drop, so thumbnails returns
before searching it. Most links on a page are text: 798 of 943 on a long
Wikipedia article, and 17,066 of 25,737 over WCXB.

Finding thumbnails takes 0.99 ms instead of 3.08 ms on the Wikipedia
article, and the whole pass 10.0 ms instead of 12.4 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The Pruner judges every link, span, div and paragraph before removing any,
so the order they are found in does not matter, and an XPath union finds
a list of tags faster than the CSS selector list did.

Removing calls to action takes 3.3 ms instead of 5.8 ms on a long
Wikipedia article.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
button? built and downcased a string of the class and id for every link,
span, div and paragraph. Matching each attribute case-insensitively
answers the same, since no Unicode letter folds to the ASCII letters of
btn, button or cta, and allocates nothing.

Checking for buttons takes 1.9 ms instead of 2.7 ms on a long Wikipedia
article (instrumented).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
paragraph_keys only collects the breakers' ancestors and the candidate
paragraphs into sets, so the order they are found in does not matter, and
an XPath union finds a list of tag names faster than a CSS selector list.

Finding paragraphs takes 3.2 ms instead of 4.4 ms on a long Wikipedia
article, and 0.58 ms instead of 0.63 ms per page on WCXB.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The redirector pattern is anchored once so most hosts fail at the first
character, but its unanchored safelinks branch made the engine try every
position of every link anyway. Matched on its own at the end, it answers
the same.

Checking for redirectors takes 0.75 ms instead of 1.29 ms on a long
Wikipedia article (instrumented).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
tr reads a non-ASCII string as characters, which on large subtrees
cost three times what a byte-level replacement does. Valid UTF-8 now
has its non-ASCII spaces replaced as byte sequences and its ASCII
spaces with tr on the bytes; other strings take tr as before.

spaces_to_blanks over every 4th WCXB page: 1.08 -> 0.69 ms/page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stats walks every node of the page twice per extraction. It now reads
each child's type once and compares it with the node type constants,
instead of calling text?, cdata? and element?, which read it again each.
A link's weight reads the href first, since most elements have none.

On a long Wikipedia article, counting the page takes 6.9 ms instead of
7.2 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Resolving a link parsed it with URI, joined it to the page's address and
cleaned its query, even when it had no query, nothing to encode and no
dot segment. A plain web URL now comes back as it is, and a plain path
from the site's root after the page's origin; Amazon's links, and a page
that is not on the web, still go through URI. Over the links of a
corpus of web pages, both ways give the same URLs, and 82% are plain.

On a long Wikipedia article, resolving its links takes 3.0 ms instead of
10.1 ms when the whole page is rendered.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Few pages carry ad-targeting attributes, yet the walk asked every
element for each of six names. One selector query over the body now
tells whether any exist, and the per-element check runs only then.

Wikipedia Coffee: advertising? 3.1 -> 1.6 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A cached signature was looked up by its class list and id joined into
one string, which every call built again. Two lookups, by class list
and then by id, find it without building anything.

Signature lookup on Wikipedia Coffee: 309 -> 246 ns per call.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Each inline node and list child was asked text?, cdata? and element?
in turn, a method call each that reads the type again. The renderer
now reads the type once and compares it with the constants, as the
character count does.

Wikipedia Coffee, full page: 33,000 predicate calls fewer.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Conditional cleaning asked every container whether it held a player,
an embed, an input, an image or a list item. Most roots hold none of
some of these, and removing containers never adds any, so the root is
asked once and its containers skip the question when the answer is no.

WCXB remove_suspicious_containers: 1.51 -> 1.38 ms/page.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Lexbor matches a selector list one selector at a time, walking the
tree once for each of the 27 hidden-candidate selectors and then
sorting the results. Wrapped in :is(), the list is matched in one walk
and returns the same elements in the same document order.

WCXB remove_hidden: 0.62 -> 0.50 ms/page; Wikipedia Coffee query
2.65 -> 1.40 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Most inline runs with a line break have no space or tab beside it, yet
each one went through a gsub that rewrote every break. That gsub now
runs only when a blank touches a break, as two anchored passes, and
doubled spaces are squeezed instead of matched by a pattern.

Output is unchanged on every corpus page. tidy takes 0.92 instead of
1.14 ms on the Wikipedia page in full mode.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
jasir-serpapi and others added 28 commits November 14, 2026 16:17
Listing what is under every table walks a nested or later table's
elements once for each table above or before it, so the time grew with
the square of the number of tables: 2.2 seconds for 800 small tables.
The outermost tables hold every table element, and listing the elements
of each takes time in proportion to the page.

An article of 1,000 sections, each a paragraph and a table, takes
1,316 ms instead of 5,165 ms (1,684 ms before the table check moved).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A heading's parent was read once per heading in it. When that parent is
inside an unclosed heading, its label is the text of the whole section,
so a page with a mistyped </h1> and hundreds of sections took seconds
where the elements were read once before. Each parent is now read from
the first heading in it, and not at all when it is read as a heading or
as a heading's child already; a heading in a heading is read as a
heading only.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Every button read the whole text of each element around it, to see
whether the element held nothing else, and listed every link of its card.
An article of paragraphs each followed by a button read the article's
text once per button: 3.3 s for 2,000 buttons and about two minutes for
8,000. Buttons are judged before any is removed, so an element's text
length and a card's links are now read once and kept for the others; the
text is compared only when the lengths match.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Reading the elements around a heading costs about as much as checking
eight elements for an id. On a page of thousands of empty headings and
one id, marking took 9 ms where checking every element took 1 ms, so
the page was slower than before with a short max_length. Where there is
a heading for fewer than eight elements, every element with an id is
read instead, as before; the two ways mark the same elements. The list
of elements is made once and also numbered.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A parent was read from its first heading, found by stepping back over
the parent's other children for every heading in it. A section of
headings each followed by a few elements was slower than reading
every element with an id. Parents read are now listed by id, so each
is read from the first heading met in it. Past eight elements sharing
an id, the step back is used again, so duplicate ids stay cheap.

The marking specs now hold the filler they meant to, so they run the
path around headings.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Parents were listed by their id, which was read and hashed for every
heading, so a long id cost its length once per heading. Past eight
parents sharing an id, every heading stepped back over its siblings
again. A heading that shares the last heading's parent is not that
parent's first, so only a heading with a new parent reads its id and
steps back to the heading before it. Siblings are stepped over once
at most, and the same elements are marked.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A parent has a label only when the heading opens its section, after
nothing but empty elements. Each new parent stepped back from its
heading to the one before it, so headings that took turns between a
section and a nested element stepped back over every gap again. The
check now also steps on from the section's first element and stops at
the first one with content, a child or a heading, so a section that
opens with text or another heading is told at once. Stepping back
still reads each element once over all headings.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a button is a card's only link kept an Array of every
link node of each card. Nested cards each held the links they share,
so peak memory doubled on pages of nested boxes that each open with a
button. The card's links are now kept as the node set the query
returns, which holds no Ruby nodes, together with the first nine
distinct keys of the links that lead elsewhere. A button that lacks
one of those keys is not the card's only link, so the set is read
again only for a button that holds them all. Output is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A card's first nine distinct link keys were read in full for the first
button in it, so a card whose first link leads elsewhere still read every
long same-page link after it. The keys are now read in order only as far
as some button has needed, and a button that holds them all goes on from
where reading stopped. No link is read twice, and the scan stops at the
first link that leads elsewhere than the button, as before.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Headings that take turns between a section and an element nested in it
made every heading tell again whether it opens the section, stepping
over the empty elements before it each time. The checks now share a
budget of one step per heading plus a sixteenth of the page's elements.
They run before anything is read, so when the budget runs out every
element with an id is read instead, which marks the same elements. No
WCXB page uses more than a twelfth of the budget.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Blocks were rendered from a list of all of an element's children, built
before the first was read, so a budget spent early still listed every
child of a long element. The children are now read through the sibling
pointers as rendering goes, and stop when the budget does.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a heading opens its parent's section stepped on from
the parent's first element. Finding that element, and the next one,
passes every text node before it, and a section whose opening text
was cut up by removed scripts or comments was passed again for every
heading: quadratic in the page.

Step over child nodes instead, text included, so each heading takes
one step per node until the walk stops. The answer is the same:
elements without content are still stepped over, and one holding an
element, or a heading, still ends the walk.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Listing the sections headings open looked up each parent's id, and
marking looked it up again. A parent whose headings take turns with
headings nested in it is listed once per turn, so a parent with many
attributes had them read twice per heading.

List only the sections a heading opens, telling each without its id,
and look the id up once, when the section is marked. Marked elements
are the same: the id is still read after the heading is marked, where
a misnested element's copies share their attributes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether a heading opens its parent's section read the text of
every element before it, to see whether it is empty. Since every
listed parent is told, with an id or without, a section without an id
had its opening text read, which marking it never needed.

Stop stepping back only at a heading or an element that holds one,
without reading text, and leave whether the elements before the
heading are empty to the label, read only for a section with an id.
That label is the one every element with an id had read before, so
the same elements are marked.

Stepping back stops at the heading before, so it passes each element
once over all headings, and only steps on are taken from the budget.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The parser repeats an element left open, and the copies share one list
of attributes, but a mark added through a copy that is not the first of
its element is lost when the first one gains an attribute. Marking in
the order of the headings could mark a later copy first, so the id was
then read as a name: "social-media" as a share bar, and every paragraph
holding a copy was removed.

Find the anchors around headings first and mark them together. Where
two of them are copies of one element, with the same id, read every
element with an id instead, which marks in document order.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Stepping back from a heading took no step from the budget, so where
each section opens with text and then many empty elements, every one
of them was passed, where the budget would have read every element with
an id instead, which is faster there.

Take every step from the budget again. The text-free check on the
elements before the heading stays.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When two elements the parser repeats shared an id, the anchors around
headings were read, then read again by every element with an id. A page
of sections whose headings hold the same anchor read each label twice.

List the elements with an id around the headings first, and read their
labels only when no two copies share an id, so each element is read
once either way.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Telling whether two listed elements were copies of one misnested element
read the id of every element the parser repeats, including those whose
label no id could match, and hashed it. A long id shared by many copies
was copied once per copy.

A later copy shares its first copy's attribute list, which that first
copy owns, so asking an id attribute for its element tells a later copy
without reading the id. Elements apart, even with the same id, own their
attributes and no longer fall back to reading every element with an id.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An attribute of a misnested element's copies names as its element the
first copy it is asked for through, not always the first copy in the
document. The check still holds, as two listed copies cannot both be
named, and a copy listed alone is marked as in document order. A spec
asks the copies for their id in reverse first.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
On Ruby 3.4 and 4.0 the first measured block allocates one object more,
and the garbage collector keeps short-lived nodes found by a query alive
past GC.start, so two specs failed there by a margin of one or two. The
signature spec now takes the fewest allocations of three runs, and the
card spec counts the nodes the pruner holds instead of the live ones.
Both still fail on the lib before the change each one covers.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The copies Lexbor leaves of a misnested formatting element share one
attribute list, and an attribute written through one copy after keys
are set (a forum's thread mark, a replaced member panel) can drop the
key of another. Stats then had no counts for that copy and returned
nil, so scoring a forum page raised NoMethodError in text_chars or
link_density, or TypeError when Boilerplate summed the characters of a
run of cards.

An element without a key is now counted each time it is asked about,
and nothing is stored for it. Pages that raised now return a result,
and every other result is unchanged. Looking up the link around an
element walks its parents instead of building the list of its
ancestors.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A keyless element was counted with the link weight of the links around
it, but not its own, so a keyless <a href> read as prose. On a forum
page whose reply link runs into the reply, that passed the reply alone
as the article and dropped the question above it. The element's own
href now counts, as it does in the walk from the root.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An element that lost its key was counted again on every lookup. Boilerplate
asks for an element's parent once per phrasing child, so a bold copy
holding N spans was walked N times, and a 150 KB forum page that used to
raise after 12 ms took 15 s. Its counts are now held, by node among the
elements from the same source position, until Stats removes nodes or a
thread is marked, the two writes that can change them. The same page takes
65 ms, and every result is the same as before.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Source positions restart in every fragment the page parses, such as
each <noscript> fallback, so unrelated elements without a key can share
one position. Searching every count held there made a page of many such
fragments quadratic.

Keep only the 16 most recently used counts for each position. A lookup
then compares at most 16 nodes, and an element asked about once per
child stays held, so the repeated-parent case stays linear.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Holding only the 16 most recently used counts per source position let
elements from parsed fragments at the same position push out an element
that is asked about once per child. Each push meant counting the whole
element again, so such pages were quadratic again.

Also hold each count in a list for its place: the key of the nearest
ancestor with one, how many elements without a key lie between, and the
source position. Nothing inside an element shares its place, so asking
about its children cannot push it out there. A lookup still searches at
most two lists of 16.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Sibling fragments, each parsed from its own <noscript>, can put any
number of elements without a key at one place: the same nearest keyed
ancestor, depth and source position. So can copies of a misnested
ancestor, which share its key. With more than HELD of them asked about
in turn, as the class-group pass does, each lookup pushed out the next
one wanted, and every element was counted again in full each time.

A place now lists its first HELD elements and pushes none out. Once
another shares it, the place holds each by its path, which no other
element in the tree has, and still finds it by node. Each element is
counted once until #remove or #forget_keyless, and a lookup still
compares at most HELD nodes in each list it reads. Only pages with
more than HELD elements at one place ever ask for a path.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When more than HELD elements without a key share a place, the ones
beyond the place's list were each found by their path. Reading a path
walks every earlier sibling at every level up to the root, so with
thousands of sibling fragments that cost more than it saved.

They are now listed first by their spot: the key of the nearest element
before or above them that has one, and the steps back or up that reach
it, which lead to the element alone from each element with that key.
Only elements sharing a full spot list, or with no key within
SPOT_STEPS, read their path. A lookup compares at most HELD nodes in
each of up to four lists.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An element without a key at a full place list was found by a nearby
key, and then by its path. Node#path counts the siblings before the
element at every level, so many such elements under one wide parent
were found in quadratic time, and a page of 800 foster-parented
fragments took 15 to 19 s, against 6 to 7 s before the path list.

A place now lists every element asked about there and pushes none out.
A list of up to 32 keeps the most recently used last, as before. A
longer one keeps its order: a search looks beside the last element
found first, since elements are mostly asked about in document order,
and the entries are also kept in NodeSets of runs of 32 and of 1,024,
which compare nodes in C. So a search compares one run of 32 in Ruby
and walks nothing in the tree. The spot and path lists go.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
@jasir-serpapi
jasir-serpapi merged commit be77ad5 into main Nov 16, 2026
4 checks passed
@jasir-serpapi
jasir-serpapi deleted the jasir/faster-extraction branch November 16, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant