Skip to content

Recover streamed and split content, and drop repeated forum text - #16

Merged
jasir-serpapi merged 34 commits into
mainfrom
wcxb-losses
Nov 9, 2026
Merged

jasir-serpapi merged 34 commits into
mainfrom
wcxb-losses

Conversation

@Alaa-abdulridha

@Alaa-abdulridha Alaa-abdulridha commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On the 2,008 WCXB pages, nokodown main (5833a27) loses by 0.1 F1 or more to Trafilatura 2.3.1 on 130 pages and to rs-trafilatura 0.1.1 on 149. Tracing where the ground-truth text disappears on each of those pages (preprocessing, boilerplate, content selection, pruning, Markdown), plus a diagnosis of the 131 pages that lose on precision, points to a handful of common causes rather than one-off sites:

  • Content hidden in the HTML but visible on the page. React and Next.js stream late content as hidden <div id="S:n"> elements at the end of the body, and a script moves them into place; without the script they were dropped (39 pages, e.g. fool.com's whole article body). Closed WAI-ARIA accordion panels hidden with display:none were dropped too.
  • Hidden copies kept as content. A hidden element holding over 30% of the page was always kept (meant for anti-flicker wrappers), so a hidden raw-Markdown copy of an article, a hidden custom-field list or a closed comparison panel swamped the output.
  • Articles split into blocks. Squarespace and Elementor sections and landing-page bands put each part of the body in its own block. Readability merges a sibling of the top candidate only on its own score, which fades with depth, so only the largest block was kept.
  • Repeated text. Forum replies quote earlier posts, forums print each member's signature and profile panel beside every post, and responsive layouts render whole sections twice (desktop and mobile copies).
  • Vocabulary gaps. Blocks named related-posts or related products entry-summary survived because a content word in the name excused them. Legal disclaimers had no name at all. "Loading Questions..." placeholders stayed, and Webflow's hidden class wasn't recognized.
  • Empty output. When the only content sat in a block whose class guesses at furniture (kress.com's mower listing in section.related-product), the page came out empty.

Change

Thirteen behavior commits, each measured on its own (table below). Then specs, the README, a speedup with byte-identical output, and one move for CPU:

  1. Streamed segments: place each hidden div#S:n into template#B:n or template#P:n, dropping the loading fallback up to its <!--/$--> marker when the page still has it. All 274 $RC/$RS instructions in the corpus pair S:n with B:n/P:n.
  2. Accordion panels: a hidden element that some aria-controls points at, and that is a region or labelled by its header, is a closed panel and stays.
  3. Sibling sections: siblings of the top candidate with few links (< 0.25) and a well-scoring block inside, or the top's class, are merged when together they hold at least 0.75 times the top's text. Alone such a sibling is often an author box or a related rail; as a set past that share, it is 93–98% ground-truth words.
  4. Webflow w-condition-invisible counts as a hiding class.
  5. Noise names: a class naming noise (related, share, ad…) is no longer excused by a content word; WordPress tag-*, category-* and has-* tokens are ignored.
  6. Repeated quotes: a Markdown quote block whose word 6-grams are at least half already in the output is dropped. Quotes of outside sources stay.
  7. Repeated sections: a run of two or more consecutive blocks of 100+ characters that all repeat earlier blocks is dropped. A single repeated block (meal plans, specs per variant, API parameters, the Mozilla fixtures) stays.
  8. Large hidden elements: one is kept only when it holds the page's <h1>, as an anti-flicker wrapper does.
  9. Disclaimers and disclosures (not the singular disclosure, which names <details> widgets) are removed. They first went into SECONDARY; commit 15 moves them to page furniture.
  10. Forum signatures: classes whose last word is signature.
  11. Forum member panels: postprofile, message-user, user-info/-details/-extras/-stats, poster-info, author-pane. The name must end the class token, so XenForo's post body message-userContent stays.
  12. Loading placeholders: "Loading " followed by an ellipsis is boilerplate.
  13. Empty-page fallback: Boilerplate now decides removals first and applies them after. If they would leave under 200 characters, the largest removal made only on a class-name guess stays, as long as it holds 200 characters and is not named after a place (sidebar, footer, menu) or an overlay (modal, cookie banner). Script-rendered shells stay empty.
  14. Specs for each behavior (12 fail on main, plus a control); the README's "What it removes" lists the new removals.
  15. Speedup: quotes are compared only up to the last quote block, as hashed 6-grams. Output is identical.
  16. Disclaimers as furniture: SECONDARY is matched against every element in several passes, and the disclaimer names made extraction about 3% slower. Boilerplate matches its vocabulary once per distinct class list, so the names move there. Quality is the same within noise.

Eight commits from the review round (see "Review round: blind judge" below) narrow rules 1, 3, 6, 9 and 11, fix two defects those rules exposed (19, 21), and correct one of the narrowings (24):

  1. Sibling sections make an article only when they hold nearly the page (amends 3): the article share is judged on the top candidate and its Readability siblings. When only the added sections reach it, they must hold 90% of the page; otherwise the page stays in page mode, as on main.
  2. Streamed segments that are mostly links stay hidden (amends 1): a segment whose links hold at least half of its visible text is left hidden, as it is without scripts. These are recommendation carousels; the segments that help have links at most a third of their text.
  3. No class names read inside Tailwind arbitrary values: lg:h-[calc(100vh-var(--nav-height))] no longer reads as "nav".
  4. Disclaimers that annotate the content right before them stay (amends 9): the footnote to a mark in the element just before it ("40–46 states*"), or a <p> continuing a prose paragraph. hasDisclaimer names the block that has a disclaimer, not the note.
  5. Page title above a section H1 that follows the opening prose: when the content's first H1 comes after 150+ characters of paragraphs and does not resemble the metadata title, the metadata title goes first and content H1s become H2s.
  6. Member panels leave the poster's name (amends 11): the panel is reduced to one line, **name**, plus the post's date when the panel holds it. A panel that names no one goes whole, and no line is written when the post opens with a by-line naming the poster (phpBB).
  7. Repeated quotes are cut to their opening instead of dropped (amends 6): the quoter's own words up to the sentence that reaches 12 words, without links, nested quotes or lead-ins, then " …". A quote with fewer than 12 more words stays whole, and a quote that repeats an earlier quote (not a post) stays whole.
  8. Footer-named disclaimers stay furniture (corrects 20): a disclaimer may stand as a note only when its name is furniture for the disclaimer word alone. The word was cut without its separators, so footer__disclaimer read as footer__, which the footer pattern does not match, and a footer disclaimer right after a prose paragraph was kept. The separators now go with the word: footer__disclaimer, footer_disclaimer, site-footer-disclaimer and disclaimer-footer stay furniture. 17 such class names sit on 18 WCXB pages; no WCXB output byte changes, and the Scrapinghub and Mozilla scores are unchanged.

Results

WCXB scored with its word-level F1 and the "without" snippets (boilerplate leaked); tokens are o200k on the Markdown. Articles are scored with the Scrapinghub shingle metric.

main this branch
WCXB dev F1 (1,497 pages) 0.8679 0.8752
WCXB test F1 (511 pages) 0.8918 0.8996
All 2,008 pages (Trafilatura 0.827, rs-trafilatura 0.826) 0.874 0.881
Boilerplate leaked, dev / test 6.57% / 7.59% 6.26% / 7.54%
Ground-truth snippets kept, dev / test 81.4% / 85.3% 81.9% / 85.9%
Tokens per page, dev / test 3,073 / 2,652 3,042 / 2,627
Scrapinghub article F1 (181 pages) 0.9451 0.9468
Mozilla article F1 (130 pages) 0.9414 0.9460
Pages losing ≥ 0.1 F1 to Trafilatura / rs-trafilatura 130 / 149 113 / 124
Summed F1 deficit on those pages, Trafilatura / rs-trafilatura 32.4 / 37.1 26.3 / 29.2

F1 by page type, all 2,008 pages:

Type Pages main this branch Trafilatura rs-trafilatura
article 1050 0.950 0.953 0.926 0.921
product 147 0.707 0.719 0.640 0.605
collection 151 0.740 0.747 0.557 0.638
listing 139 0.737 0.745 0.676 0.636
forum 164 0.778 0.811 0.722 0.774
documentation 133 0.935 0.935 0.922 0.939
service 224 0.838 0.849 0.780 0.740

Per commit, each against the one before (commits 1–16; commits 17–24 are measured in the review round section: 17–23 each alone on top of commit 16 and all together, 24 against 23). CPU is an in-process A/B, alternating page by page, min of 3 runs, on every 16th WCXB page (126). The machine's load average was 200–400, so ±1.5% is noise.

# Change dev F1 test F1 dev / test leaked dev / test tok Scrapinghub Mozilla WCXB up / down (≤ −0.05) CPU total
main 0.8679 0.8918 6.57% / 7.59% 3073 / 2652 0.9451 0.9414
1 streamed segments 0.8691 0.8918 6.59% / 7.59% 3085 / 2652 0.9451 0.9414 7 / 3 (2) 0.995
2 accordion panels 0.8695 0.8918 6.59% / 7.59% 3087 / 2652 0.9451 0.9414 4 / 0 (0) 0.985
3 sibling sections 0.8699 0.8948 6.51% / 7.59% 3084 / 2663 0.9451 0.9414 15 / 10 (5) 1.006
4 w-condition-invisible 0.8699 0.8948 6.51% / 7.59% 3084 / 2663 0.9451 0.9414 1 / 0 (0) 1.002
5 noise names 0.8699 0.8948 6.49% / 7.59% 3082 / 2663 0.9451 0.9415 7 / 3 (2) 1.011
6 repeated quotes 0.8709 0.8958 6.49% / 7.59% 3068 / 2646 0.9451 0.9415 44 / 31 (6) 0.998
7 repeated sections 0.8712 0.8972 6.49% / 7.59% 3063 / 2640 0.9451 0.9415 25 / 10 (2) 1.005
8 large hidden elements 0.8718 0.8972 6.49% / 7.59% 3052 / 2640 0.9467 0.9460 4 / 2 (2) 1.010
9 disclaimers 0.8720 0.8980 6.47% / 7.59% 3047 / 2634 0.9468 0.9460 26 / 6 (2) 1.032
10 signatures 0.8729 0.8990 6.43% / 7.59% 3044 / 2630 0.9468 0.9460 35 / 3 (0) 1.015
11 member panels 0.8736 0.9002 6.25% / 7.54% 3035 / 2616 0.9468 0.9460 42 / 3 (1) 0.992
12 loading placeholders 0.8737 0.9002 6.24% / 7.54% 3035 / 2616 0.9468 0.9460 5 / 0 (0) 0.982
13 empty-page fallback 0.8743 0.9006 6.24% / 7.54% 3036 / 2616 0.9468 0.9460 2 / 0 (0) 0.993
15 speedup 0.8743 0.9006 6.24% / 7.54% 3036 / 2616 0.9468 0.9460 0 / 0 (0) 0.999
16 disclaimers as furniture 0.8745 0.9002 6.24% / 7.54% 3037 / 2618 0.9468 0.9460 8 / 5 (3) 0.987

CPU of the branch at commit 16 against main, 502 WCXB pages, run in both orders:

Order Total CPU Median page
main first +1.3% +1.6%
branch first +0.7% +1.8%

After the review round, benchmark/speed.rb --runs 3 on the 511 WCXB test pages, one checkout at a time. The review commits cost nothing measurable:

Checkout Median page, per run p90, per run
main 18.3 ms 53.7 ms
commit 16 (d7c3e6b) 18.4, 18.4, 18.8 ms 48.3, 51.5, 53.0 ms
this branch 18.0, 18.2 ms 49.4, 51.4 ms

A first run of this branch, at 19.6 / 58.4 ms, ran while other jobs loaded the machine and is left out.

The pages that cost most are pages that now extract more content: eBay dev/0614, Product Hunt dev/2914, fool.com dev/4503. Pages that now drop a large hidden block get faster (dev/0637: 462 → 159 ms).

The pages named in the investigation:

Page main branch Note
dev/4247 kress.com listing 0.00 1.00 empty-page fallback
dev/4503 fool.com article 0.08 0.92 streamed segments
dev/0628 REI product 0.83 0.84 no more empty "Questions & Answers" heading over "Loading Questions..."
dev/4677 techgearlab, dev/4355 arc.dev 0.04, 0.20 unchanged ground truth is only the intro paragraph; the reviews and job list are kept
dev/1572 redfin 0.17 0.16 listing text is in aria-hidden carousel slides; not changed
dev/0296 law blog 0.24 unchanged Readability's class weights (elementor-widget-container −25, elementor-post__text +25) pick a teaser grid

Pages that got worse

180 WCXB pages rise and 65 fall (by more than 0.0005); 13 fall by 0.05 or more. At commit 16 it was 183, 57 and 17: commits 17 and 18 bring dev/0625, dev/4332 and dev/4477 back to main, and commit 23 lifts dev/0545 to −0.02.

Page F1 Commit Reason
dev/2914 Product Hunt 0.649 → 0.411 1 Streamed rails (yesterday's, last week's and last month's top products) are rendered below today's list. Links are 0.44 of their text, under commit 18's "mostly links" cut. The ground truth has only today's list (92 tokens). It was 0.066 at commit 16.
dev/4058, test/4058 Harvard Health 0.970 → 0.649 3 "About the Author", "About the Reviewer" and a disclaimer are merged as sections beside an article that already passes the share; the ground truth leaves them out.
dev/2888 eBay 0.677 → 0.370 8 Removing the large hidden dialog makes the seller-feedback list hold most of the page, so Boilerplate keeps it as dominant.
dev/4654 Merrill calculator 0.678 → 0.481 8 A collapsed "Methods" section (39 paragraphs, no title) that the ground truth keeps is now removed.
dev/0341 Yosemite webcams 0.981 → 0.805 7 The same credit and donation paragraphs follow every webcam, and the ground truth repeats them.
dev/0251 HRSA 0.840 → 0.677 5 view--hrsa-promoted-blocks, the site's own promo cards, which the ground truth keeps.
dev/4184 Consumer Reports 0.983 → 0.865 5 A related-articles rail the ground truth keeps.
dev/0135 0.910 → 0.832 3 The sections hold nearly the whole page, so it still becomes an article, and article cleaning removes part of a band. No share threshold separates it from dev/3027, whose judged win rests on the same cleaning.
dev/4564, dev/0544, test/5072 −0.06 to −0.07 6, 7, 23 Forums whose ground truth keeps full quoted copies of earlier posts; they now keep the opening.
dev/0112 0.978 → 0.894 9 A disclaimer the ground truth keeps, with no mark or prose paragraph before it.

Tried and dropped

All measured on the full benchmark:

  • Widening the top candidate to ancestors that hold more prose: 185 pages up, 579 down.
  • Letting no class token that names a place carry a content word (sidebar-main): dev/4091 −0.91 and test/5004 −0.73, whose content wrappers are named content-sidebar.
  • Removing every repeated block: +4.4 F1 summed on WCXB, but it breaks meal plans (a Scrapinghub page) and the Mozilla fixtures.
  • No class weight on the <main> region root: test F1 −0.0012.
  • Judging siblings through single-child wrappers, to catch the GearLab intro: 27 up, 48 down, both article corpora down.

Left for later

  • Product pages: the 0.9 share keeps tabs, FAQs and recommendation carousels, or the top is main or a whole tab widget.
  • Class weights picking the wrong top: Elementor's widget, and main +25.
  • Forum threads without forum metadata: test/5104, Slashdot dev/0570.
  • Carousel slides marked aria-hidden.
  • Questionable ground truth: about 2.5 F1 of the remaining deficit sits on pages whose ground truth is intro-only, truncated or inconsistent; 30 of 44 sampled ground-truth files are rs-trafilatura drafts.
  • Stack Overflow post signatures (dev/4309) and sections that hold the whole page (dev/4017), from the review round below.

Review round: blind judge

Blind pairwise judges (Claude, given the reference and both outputs, scoring completeness, noise and formatting) compared main with main + #16 + #18 + #19 on the 50 of 210 panel pages where the two differ. The open PRs together won 29 and main 17, with 4 ties (mean overall score 6.82 against 6.03). But on the 32 pages that only this PR changes, this PR won 13 and main 17 (2 ties), although it raised WCXB F1. Every page main won came from this PR alone: #18 and #19 equal main there.

What the judges found, and the cause

Rule What went wrong Judged pages Fix
6 repeated quotes Dropping a quote that repeats an earlier post left replies that answer nothing and "X posted..." lines with no quote after them (21 such lines on 4 pages; main has 2). A quote of an earlier quote (a testimonial shown twice, nginx examples, a customer quote) was dropped as well. 0574, 0557, 0558, 4564, 0576, 0084, 0698 23
11 member panels Removing the whole panel removed the only poster name: dev/0520 kept 0 of 15 names (main kept 13, buried in stats). 0520 22
9 disclaimers A footnote to a mark in the table above it and a "Note:" paragraph in the body were classed as fine print, and hasDisclaimer read as a disclaimer. 0054, 0642 20
9 disclaimers Once the affiliate disclosure no longer lifted the title block, a mid-page section H1 stood as the page's only H1. 4064 21
3 sibling sections Where the top candidate falls short of the article share, the sections switched page mode to article mode, which drops every band they leave out: hero, intro, claims, call to action. 0247, 4332 17
1 streamed segments Streamed recommendation carousels diluted a product block, so the whole <main> was rendered. On another shop, the streamed description lowered the configurator's share and exposed lg:h-[calc(100vh-var(--nav-height))] read as "nav", so the configurator was removed. 0625, 0602 18, 19
10 signatures Stack Overflow's post signature holds the "asked / answered date" line, and it went with the signature. 4309 not fixed
3 sibling sections The sections hold nearly the whole page, and article cleaning drops the price and the shipping line. 4017 not fixed

The 17 pages main won, before and after

WCXB F1 against the reference.

Page main commit 16 now Output now
dev/0574 GameFAQs 0.610 0.687 0.708 The three quotes are back whole (12–13 words), so each "X posted..." has its quote. One search-box line, "What do you need help on? Cancel X", has come through since commit 10 and remains.
dev/0557 Flarum 0.999 0.958 0.992 All 12 point-by-point excerpts are back; the long ones as their opening sentence.
dev/0558 MyBB 0.856 0.865 0.888 The three "StarredSkies Wrote:" quotes are back; the four-item list as its first item.
dev/4564 permies 0.947 0.859 0.874 Partly: the five quotes are back as attribution and opening. The reference keeps the full copies.
dev/0576 Nairaland 0.963 0.932 0.938 Partly: the quote is back as its opening. A 4-word line unique to the quote ("Just a quantum compass") is still lost.
dev/0084 1.000 0.983 1.000 Identical to main: the second testimonial is back.
dev/0698 Stripe 0.606 0.612 0.606 Identical to main: the repeated customer quote is back.
dev/0520 XenForo 0.858 0.936 0.948 15 of 15 posts have a **name** line; the member stats stay out.
dev/0054 move.org 0.865 0.882 0.878 The footnote "*Colonial Van Lines serves 40 states..." is back under the table carrying "40–46 states*". The other 10 disclaimer blocks stay out.
dev/0642 Green Roads 0.887 0.885 0.887 Identical to main: the state-law note is back.
dev/4064 PCMag 0.929 0.936 0.935 Opens with "# The Best Ergonomic Mice for 2026" and the intro; "Deeper Dive" is an H2 again.
dev/0247 landing page 0.876 0.830 0.876 Identical to main: H1, intro, claim and call to action are back.
dev/4332 CSRankings 0.890 0.781 0.890 Identical to main.
dev/0625 Zalando 0.703 0.245 0.703 Identical to main: about 3,800 tokens back to 320.
dev/0602 UPLIFT Desk 0 0 0 The configurator is back, with the streamed description after it. The reference is empty.
dev/4309 Stack Overflow 0.951 0.968 0.968 Not fixed. Sparing a signature that holds [itemprop~=author] would let commit 22 write "name · asked Jan 4, 2024 at 11:30"; left for a follow-up because it touches about 20 Stack Exchange pages.
dev/4017 ProSource 0.924 0.935 0.935 Not fixed: "$29.99" and the shipping line are still dropped. Never letting sections make a page an article fixes it, but gives back dev/3027, a page this PR won, and the gains on 3029, 4379, 0302, 0603 and 4705 (about 0.13 summed F1).

15 of the 17 are fixed or improved (4564 and 0576 in part); 4309 and 4017 are not.

The 23 pages this PR won

11 are byte-identical to commit 16. The other 12 change only by quote openings and name lines, plus four restored "See SUMMARY.md" links on dev/0498 (0.993 → 0.998). Of those 12, all stay above main on F1 except dev/0545 (main 0.924, commit 16 0.867, now 0.903), which now differs from main only in its quote, cut to one sentence; all but 0498 stay below main on tokens. Four forum pages give back part of their F1 gain because their references leave the quotes out: dev/0538 0.968 → 0.896 (main 0.802), 0537 0.971 → 0.934, 0547 0.486 → 0.461, 0579 0.742 → 0.720. On 0538 the replies are tied to the numbered questions again, at 5,629 tokens against main's 7,006.

Numbers

Each fix alone on top of commit 16 (d7c3e6b), and all together:

Commits dev F1 test F1 dev / test forum dev / test tokens Pages changed (up / down ≥ 0.02)
commit 16 0.8745 0.9002 0.7943 / 0.8559 3037 / 2618
17 sibling sections 0.8747 0.9002 0.7941 / 0.8559 3037 / 2618 4 (3 / 0)
18, 19 streamed segments 0.8751 0.9002 0.7943 / 0.8559 3033 / 2618 6 (3 / 0)
20, 21 disclaimers, title 0.8745 0.9002 0.7943 / 0.8559 3037 / 2619 33 (0 / 0)
22 member panels 0.8746 0.9000 0.7955 / 0.8534 3038 / 2620 45 (2 / 1)
23 repeated quotes 0.8744 0.8998 0.7924 / 0.8522 3043 / 2624 75 (12 / 21)
17–24 together 0.8752 0.8996 0.7931 / 0.8498 3042 / 2627 132 (18 / 22)

By page type, main / commit 16 / now:

Type dev test
article 0.9496 / 0.9524 / 0.9524 0.9503 / 0.9538 / 0.9537
collection 0.7218 / 0.7292 / 0.7292 0.8021 / 0.8066 / 0.8064
documentation 0.9319 / 0.9321 / 0.9325 0.9419 / 0.9415 / 0.9416
forum 0.7600 / 0.7943 / 0.7931 0.8191 / 0.8559 / 0.8498
listing 0.7280 / 0.7321 / 0.7367 0.7578 / 0.7652 / 0.7652
product 0.6789 / 0.6865 / 0.6914 0.8253 / 0.8342 / 0.8348
service 0.8381 / 0.8475 / 0.8483 0.8381 / 0.8526 / 0.8526
overall 0.8679 / 0.8745 / 0.8752 0.8918 / 0.9002 / 0.8996
  • Gain over main: dev +0.0073 (commit 16: +0.0066), test +0.0078 (commit 16: +0.0084). Test forum gives back 0.0061, because its references mostly leave poster names and quoted copies out.
  • Snippets kept / leaked: dev 0.814 / 0.818 / 0.819 and 6.57% / 6.24% / 6.26%; test 0.853 / 0.859 / 0.859 and 7.59% / 7.54% / 7.54%. Empty pages: dev 19 / 17 / 17, test 6 / 5 / 5.
  • Pages moved by 0.02 or more against main: dev 78 up and 15 down (commit 16: 79 and 24), test 37 up and 2 down (commit 16: 41 and 2).
  • Scrapinghub F1 0.9468 and Mozilla 0.9460, as at commit 16. On Scrapinghub only one page changes (recall 0.981 → 1.0: a views note its reference keeps); Mozilla is identical page by page.
  • Commit 24 changes no output on the 2,008 WCXB pages: they are byte-identical to commit 23 (59ecec4).
  • The fixes are nearly independent: on 4 forum pages the combined score differs from the sum of the single-fix changes by more than 0.003 (at most −0.006, where a name line and a shortened quote meet).

Combined with #18

Merging #18 on top (scratch branch, not pushed): on dev/0520 every post has one name line. dev/0558 and dev/4309 get no attribution line from either PR (conditional cleaning removes MyBB's poster name, on main too; 4309 as above). On dev/0523 every post gets #18's "name · date" line and this PR's "name" line (20 pairs). #18's PostAttribution#mark has to remove the member panels inside each post it attributes, a 9-line change on #18's side; with it there are no duplicates on the 8 forum pages checked.

Blind judge, second round

The same judges, on the same 210-page panel, compared main with main + this PR (213130d) + #18 + #19. The outputs differ on 49 pages: the PRs won 42 and main 3, with 4 ties (mean overall score 7.43 against 5.79; the first round was 29 to 17, 6.82 against 6.03). On the 31 pages that only this PR changes, it won 26 and main 3, with 2 ties (first round: 13 to 17). main still wins:

  • dev/4309: the Stack Overflow "asked / answered" lines go with the signature (rule 10, not fixed; the reference leaves them out).
  • dev/4017: the price and shipping line (rule 3, not fixed; the reference has no price).
  • dev/0576: "Just a quantum compass", a line of its own inside a quote now cut to its opening (rule 23).

Review notes left open

Two reviews of commits 17–23 found one blocking defect, fixed in commit 24. The non-blocking notes:

  • Quote shortening (23) runs on every page type, so article pull quotes that commit 16 dropped come back: whole, as on main (dev/0324), or as an opening (dev/0496, −0.006 against commit 16).
  • test/5115: a name line (22) moves one thread's peer share from 0.751 to 0.748, under the 0.75 cut, and the short reply "Border Terrier or lurcher!" is lost.
  • NAME_CLASS and DATE_CLASS in member_panel.rb match broadly on synthetic markup: a rank-name span can be read as the name, a timezone span as the date. On WCXB every name found is a poster.
  • A shortened opening can cut through a table row or leave italics open, and a reply whose own words are only a lead-in becomes "> Sam said: …". Not seen on WCXB.
  • A streamed segment that is a link grid and also the page's only content (a category grid, an API index) stays hidden, as on main.
  • prose_note? keeps a <p class="disclaimer"> right after an 80-character paragraph, where site fine print often sits. WCXB has 4 such notes; one is in its reference.
  • A long figcaption counts as opening prose, so a real headline after it can be demoted under the metadata title (21).

How it was tested

  • bundle exec rspec: 261 examples, 0 failures on Ruby 2.7.8 with nokolexbor 0.6.4 and on Ruby 3.4.5 with 0.6.4 and 0.8.0 (commit 16's 241 examples also passed on Ruby 2.7.8 with 0.8.0). The 12 behavior specs of commits 1–13 fail on main; the control (a quote of text not on the page stays) passes on both. Of the 21 specs added in the review round (one old quote spec replaced), 16 fail on commit 16 and 5 are guards (fine print still removed, footer-named fine print still removed after a prose paragraph, a stats-only panel still removed whole, a phpBB by-line still the only name, a headline below a photo caption kept). The footer-named guard fails at commit 23.
  • All 2,319 benchmark pages (WCXB dev and test, Scrapinghub, Mozilla) extracted after every commit and compared with the commit before: the scores above, every changed page reviewed by F1 and token delta, 0 errors. The speedup commit changes no output byte. The review-round fixes were each run alone on top of commit 16, then together.
  • After the review round, the outputs of the 17 pages main won and the 23 pages this PR won were read and compared with main and commit 16.
  • In-process CPU A/B as above, per commit and for the whole branch in both orders; benchmark/speed.rb for the review round.

🤖 Generated with Claude Code

Alaa-abdulridha and others added 17 commits October 19, 2026 12:16
React and Next.js stream the parts of a page that render late as hidden
<div id="S:n"> elements at the end of the body, and an inline script moves
each one into the <template id="B:n"> or <template id="P:n"> left in its
place. Without scripts those divs stayed hidden and were removed, so on 39
WCXB pages the article body, product details or FAQ never reached the
output (fool.com, render.com, wayfair). Every $RC and $RS instruction in
the corpus (274) pairs S:n with B:n or P:n, so the slot is found by number.
The loading fallback after a B: template is dropped up to its <!--/$-->
marker when the page still has it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An accordion panel that a control on the page opens (its id is in some
aria-controls) and that is labelled by its header or is a region, as in
the WAI-ARIA accordion pattern, is content even while collapsed with
display:none or the hidden attribute. Such panels hold 92-99% ground-truth
words across WCXB, where only panels named like an accordion were kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Readability merges a sibling of the top candidate only when the sibling's
own score clears the threshold, but scores fade with depth, so a page
split into blocks (Squarespace and Elementor sections, landing-page bands)
kept one block: 15 WCXB pages lost most of their text this way.

A sibling with few links and a well-scoring block inside it, or with the
top's class, is a section of the same body. Alone such a sibling is often
an author box or a related rail, but when these siblings together hold at
least three quarters as much text as the top candidate, they are the rest
of the page: across WCXB that set is 93-98% ground-truth words.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Webflow marks CMS elements whose visibility condition fails with
w-condition-invisible, which its stylesheet sets to display:none. 26 WCXB
pages carry it, some on a gated duplicate of the whole article.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block named after page furniture survives when its classes also contain
a content word such as "post" or "entry" and it holds 200 characters:
"content-sidebar" can wrap the article. Noise names are not places, so
"related-posts", "related-pages" and "related products entry-summary"
were kept as content; across WCXB such blocks hold 6.5% ground-truth text.
Taxonomy tokens that WordPress puts on posts (tag-*, category-*) and
has-* modifiers are not names of the block and are ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum replies quote earlier posts (XenForo bbCodeBlock--quote, Invision
ipsQuote, phpBB "wrote:" blocks), so the same text is rendered again
under every reply: a quarter to a third of the output on many threads. A
quote block whose word 6-grams mostly occurred earlier in the Markdown is
a copy and is dropped. Quotes of outside sources are new text and stay.

On WCXB this changes 76 pages, 75 of them threads: forum F1 rises from
0.760 to 0.773 (dev) and 0.819 to 0.829 (test), and output shrinks by 30k
tokens. The 31 pages that drop are threads whose ground truth keeps the
quoted copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Responsive layouts render a section once for desktop and once for mobile
(d-none d-md-block / d-md-none, hidden-md-down / hidden-md-up), shops
render a grid and a list view, and forums print a post's body twice; both
copies are visible to the Preprocessor, so the Markdown repeated them.

A run of two or more consecutive blocks of 100+ characters that all
repeat earlier blocks is such a copy and is dropped. A single repeated
block is kept: meal plans, specs per variant and API parameters repeat
one block legitimately, and the Mozilla fixtures repeat paragraphs.
On WCXB 38 pages change (25 up, 10 down) and output shrinks by 10k
tokens; the article corpora are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hidden element holding over 30% of the page is kept because
anti-flicker snippets hide the whole page until scripts run. Such a
wrapper holds the page's <h1>. A large hidden block beside a visible
title is not one: a hidden raw-Markdown copy of the article (Zapier),
BigCommerce's hidden custom-field list, a closed product-comparison panel.
They were kept and doubled or swamped the output.

Scrapinghub F1 0.9451 -> 0.9467, Mozilla 0.9415 -> 0.9460; on WCXB 4
pages rise (dev/0637 +0.54, dev/0604 +0.34, dev/4258 +0.33) and 2 fall: a
collapsed "Methods" section the ground truth keeps (dev/4654), and an
eBay page where removing the hidden dialog left seller feedback dominant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Legal footnotes, affiliate disclosures and "rates subject to change"
notes sit inside the main region, often in the same rich-text widget as
the content, and were rendered after it, sometimes as a third of the
output. Named disclaimers or disclosures, they are now secondary like
comment sections and author bios: they do not count against the article
and article cleaning drops them. The singular "disclosure" names the
<details> widget pattern and stays content, and WordPress category-* and
tag-* terms on a post are not names of a block.

On WCXB 35 pages change (28 up, 7 down), output shrinks by 11k tokens,
and the article corpora are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum software renders each member's signature under every post
(XenForo message-signature, vBulletin and phpBB signature, Invision
signature), so the same lines repeat through a thread, and on threads with
long signatures they outscored the posts in Readability scoring. A class
or id whose last word is "signature" now names page furniture; names
that only start with it, such as signature-dish or signature-collection,
do not.

On WCXB 39 pages change, 36 up and 3 down: forum F1 rises from 0.773 to
0.785 on dev, and dev/2201 from 0.35 to 0.87.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum software prints a panel of member details next to every post:
avatar, rank, join date, message and reaction counts, location (XenForo
message-user and message-userExtras, phpBB postprofile, Invision
cAuthorPane, vBulletin userinfo). Repeated through a thread it was a
sixth of the output on many forums. Classes whose last words name such a
panel are now page furniture; the token must end there, so XenForo's post
body, message-userContent, is not one.

On WCXB 48 pages change, 42 up and 3 down: forum F1 rises from 0.785 to
0.794 on dev, output shrinks by 20k tokens, and the article corpora are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Widgets that load their content with scripts leave a placeholder such as
"Loading Questions..." or "Loading reviews…" in the HTML, and only a
bare "Loading" was recognized: REI product pages ended with an empty
"Questions & Answers" heading over "Loading Questions...". One or two
words between "Loading" and an ellipsis now count too; without the
ellipsis ("Loading dock") the text stays. With the placeholder gone the
heading has no section left and is dropped as before.

On WCXB and Scrapinghub 7 pages change, all up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…page

Boilerplate removes semantic chrome (nav, page header and footer, chrome
roles) and blocks whose class names guess at furniture. On kress.com's
mower listing the products sit in section.related-product, so the guesses
left 178 characters and the page came out empty.

Removals are now decided first and applied after. When they would leave
under 200 characters, the largest guessed removal stays, provided it holds
200 characters itself and is not named after a place on the page (sidebar,
footer, menu) or an overlay (modal, cookie banner). Script-rendered shells,
whose guesses are drawers, sidebars and sign-up buttons, stay empty.

On WCXB this changes 2 pages: dev/4247 0.00 -> 1.00, test/5301 0.00 ->
0.20.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One example per behavior added on this branch: React streamed segments,
closed accordion panels, Webflow's hidden class, large hidden copies
beside the title, articles split into blocks, disclaimers, noise names
with content words, forum signatures and author panels, the empty-page
fallback, repeated quotes and sections, and loading placeholders. Each
fails on main; a quote of text that is not on the page stays, on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quotes are compared only up to the last quote block, word 6-grams are
kept as hashes instead of joined strings, and pages without a repeated
long block skip grouping runs. Output is unchanged; Markdown cleanup on
111 WCXB pages took 16 ms less.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SECONDARY is matched against every element's classes in several passes
(scoring, the article's share, article cleaning), so the disclaimer
alternative made extraction about 3% slower in an in-process A/B on 126
WCXB pages. Boilerplate matches its vocabulary once per distinct class
list, and a disclaimer is furniture on any page, so the names move there.
Quality is unchanged within noise: 13 WCXB pages change, dev F1 +0.0002,
test -0.0004, the article corpora are identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alaa-abdulridha and others added 8 commits October 28, 2026 15:58
…page

Sibling sections extend an article split into blocks. On a page whose top
candidate falls short of the article share they also switched page mode to
article mode, which drops every band they leave out: a landing page lost its
intro, a claim and its call to action, a ranking page its introduction.

The share is now judged on the top candidate and its Readability siblings.
When only the sections reach it, they must hold 90% of the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
React streams last what waits on another service, on shops mostly
carousels of recommended products. Placed into main, they pushed the
product block of a product page below the share content selection needs,
and the whole page was rendered. Segments that help (article bodies,
product details, FAQ answers) have links at most a third of their text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lg:h-[calc(100vh-var(--nav-height))]" read as "nav" furniture, so a
product configurator was dropped once streamed content made it less than
half of the page. The utility name around the brackets is still read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block named only as a disclaimer stays when it is the footnote to a mark
in the element just before it ("40-46 states*") or a paragraph continuing
the prose paragraph before it. A modifier such as hasDisclaimer names the
block that has a disclaimer, so it no longer counts as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without its title block, for example once an affiliate disclosure no longer
lifts the block's score, the content's first H1 can be a section heading
below the opening paragraphs, and it stood as the page's only H1. Such an
H1 is now demoted under the metadata title. A headline after a caption or a
promo line stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing the member panel whole also removed the only name on many
posts: XenForo 1, phpBB and vBulletin threads lost who wrote each reply
(dev/0520 kept none of its 15 names). The panel is now reduced to one
line, "**name**", with the post's date when the panel holds it; avatars,
ranks, join dates and counts still go. A panel that names no one goes
whole, and no line is written when the post opens with a by-line that
names the poster, as phpBB's does.

On WCXB against PR 16 at d7c3e6b, dev forum F1 rises from 0.7943 to
0.7955 (29 pages change, 21 up) and test forum F1 falls from 0.8559 to
0.8534, as its references mostly leave names out. The article corpora
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping a reply's quote of an earlier post left replies that answer
nothing, "X posted..." lines with no quote after them, and lost a
testimonial a page shows twice. A quote whose 6-grams mostly repeat
earlier posts now keeps the quoter's own opening, up to the sentence that
reaches 12 words (lead-ins like "Sam said:", nested quotes and links left
out), then an ellipsis. A quote with fewer than 12 more words stays whole.
Only posts count as the source, not earlier quotes.

On WCXB against PR 16, dev F1 0.8745 -> 0.8744 and test 0.9002 -> 0.8998.
Forum output keeps about 60% of the rule's token savings (dev 2141 tokens
per page, 2067 when dropping, 2244 when keeping quotes whole).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A disclaimer can be kept as a note only when its name is furniture for the
disclaimer word alone. The word was cut without its separators, so
"footer__disclaimer" read as "footer__", which the footer pattern does not
match, and the footer's fine print after a prose paragraph was kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasir-serpapi and others added 2 commits November 9, 2026 15:17
The segment's id went into a CSS selector as is, so an id such as "S:1']"
or "S:\" raised Nokolexbor::Lexbor::UnexpectedDataError and the whole
conversion failed. The slots are now read once into a hash keyed by
number. A slot that sat inside a fallback removed for an earlier segment
is skipped, as the selector did when it no longer found it.

All 339 segment ids on WCXB's 39 streamed pages are hex, so the output
there does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasir-serpapi and others added 7 commits November 9, 2026 15:38
A Suspense fallback can hold boundaries of its own. The fallback was cut
at the first "/$" comment, which closes the nested boundary, so the rest
of the outer fallback ("Still loading.") stayed beside the streamed
content. The cut now counts "$", "$?" and "$!" comments in and "/$"
comments out, and stops at the one that closes the slot's boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pattern for an arbitrary value let "[" into its contents, so on Ruby
2.7, whose regexp engine has no memoization, a class with a run of
unclosed "[" was rescanned from each one. 30,000 of them took 8.05 s
against 1.0 ms on main; they now take 4.1 ms. Brackets inside the value
are allowed one level deep, possessively, so a variant such as
"[&:not([hidden])_.share-row]:flex" is stripped whole rather than from
its inner "[" on, which left "share-row" to read as a share widget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sentence-end pattern could start inside a run of ".", "!", "?" or
"…" and backtrack through it, so on Ruby 2.7 a quoted paragraph with a
long run of dots was rescanned from each one: 30,000 dots took 11.98 s
against 4.6 ms on main. The match now starts at the first mark of a run
and its quantifiers are possessive; it takes 7.3 ms. It finds the same
sentence ends as before on 200,000 random strings of marks, quotes,
brackets, letters and spaces, on Ruby 2.7, 3.4 and 4.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To find where a quote's opening ends, the whole line up to each sentence
end was cut out and its words counted again, so a line of many short
sentences was read once per sentence, on any Ruby: 64,000 "! " took
over 30 s against 17 ms on main. The head now grows a sentence at a
time and only the new text is counted, once any bracket or code span in
it closes, so a link that spans sentences is still read whole; it takes
73 ms. On 1,000,000 random lines it ends the opening where it did
before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any class or id ending in "signature" was noise, which also removed a
function's signature in API docs (TypeDoc tsd-signature, Javadoc
member-signature, YARD's h3.signature heading itself) and a post whose
body is marked as having one ("post-text has-signature"): with no
content word in its name and under 200 characters, nothing saved it.

A signature now needs a post around it: a name that, without the
signature word, matches post, message, comment, reply, thread, topic,
forum, answer and the like, or an ancestor named so. "has-signature" and
"with-signature" are modifiers and never match. Of the 515 signature-named
elements on WCXB, all but 16 have that context, and those 16 are icons,
empty spans and wiki headings about PGP signatures. Only three pages
change, each keeping an artist's or a CEO's signature image; F1 is
unchanged. On Mozilla, lemonde-1 keeps its "Le Monde | date | Par"
byline (p.bloc_signature) again, as main does: 0.994 -> 0.984 precision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A large hidden element is kept as an anti-flicker wrapper only when it
holds every h1 on the page. Many themes mark up the site logo as an
<h1> in the header, outside that wrapper, so the whole page was removed
and the output was just the logo. An h1 in a nav, an image-only h1, or
one whose text is a link to the home page ("/", "/en/", the site root)
no longer counts on either side. A linked h1 that goes elsewhere still
counts.

No WCXB page changes: the h1s outside its large hidden blocks are real
titles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WooCommerce puts the product's category, tag and brand terms on the
product element ("product_cat-cookies", "product_tag-social-gifts"),
as WordPress does tag-* and category-* on a post. Those were already
ignored when deciding whether a noise name is excused by a content
word; the WooCommerce ones were not, so a product filed under
"cookies" or tagged "social gifts" was removed as a cookie banner or a
share bar, and a lesser block took its place as the content.

No WCXB page changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jasir-serpapi
jasir-serpapi merged commit 0973192 into main Nov 9, 2026
4 checks passed
@jasir-serpapi
jasir-serpapi deleted the wcxb-losses branch November 9, 2026 13:26
jasir-serpapi added a commit that referenced this pull request Nov 9, 2026
lib/nokodown.rb conflicted only on the requires: both member_panel (#16)
and post_attribution are loaded.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
jasir-serpapi added a commit that referenced this pull request Nov 9, 2026
On XenForo themes that print the name in the member panel and again in
the post header (dev 0523), PostAttribution read the name from the
header, and #16's MemberPanel reduced the panel beside it to a second
"**name**" line: every post opened with two lines, 20 pairs on 0523.

The member panels inside an attributed post, outside its body, are now
part of what the line replaces. Boilerplate still reduces them, so the
selection does not change, and the line they leave goes with the
headers when the content is selected.

WCXB: only dev 0523 changes, 0.627 -> 0.632; no page has a pair left
but dev 2797, where the post under the first line was pruned (as on
main).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants