Repository navigation
Recover streamed and split content, and drop repeated forum text - #16
Merged
Merged
Conversation
React and Next.js stream the parts of a page that render late as hidden <div id="S:n"> elements at the end of the body, and an inline script moves each one into the <template id="B:n"> or <template id="P:n"> left in its place. Without scripts those divs stayed hidden and were removed, so on 39 WCXB pages the article body, product details or FAQ never reached the output (fool.com, render.com, wayfair). Every $RC and $RS instruction in the corpus (274) pairs S:n with B:n or P:n, so the slot is found by number. The loading fallback after a B: template is dropped up to its <!--/$--> marker when the page still has it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An accordion panel that a control on the page opens (its id is in some aria-controls) and that is labelled by its header or is a region, as in the WAI-ARIA accordion pattern, is content even while collapsed with display:none or the hidden attribute. Such panels hold 92-99% ground-truth words across WCXB, where only panels named like an accordion were kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Readability merges a sibling of the top candidate only when the sibling's own score clears the threshold, but scores fade with depth, so a page split into blocks (Squarespace and Elementor sections, landing-page bands) kept one block: 15 WCXB pages lost most of their text this way. A sibling with few links and a well-scoring block inside it, or with the top's class, is a section of the same body. Alone such a sibling is often an author box or a related rail, but when these siblings together hold at least three quarters as much text as the top candidate, they are the rest of the page: across WCXB that set is 93-98% ground-truth words. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Webflow marks CMS elements whose visibility condition fails with w-condition-invisible, which its stylesheet sets to display:none. 26 WCXB pages carry it, some on a gated duplicate of the whole article. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block named after page furniture survives when its classes also contain a content word such as "post" or "entry" and it holds 200 characters: "content-sidebar" can wrap the article. Noise names are not places, so "related-posts", "related-pages" and "related products entry-summary" were kept as content; across WCXB such blocks hold 6.5% ground-truth text. Taxonomy tokens that WordPress puts on posts (tag-*, category-*) and has-* modifiers are not names of the block and are ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum replies quote earlier posts (XenForo bbCodeBlock--quote, Invision ipsQuote, phpBB "wrote:" blocks), so the same text is rendered again under every reply: a quarter to a third of the output on many threads. A quote block whose word 6-grams mostly occurred earlier in the Markdown is a copy and is dropped. Quotes of outside sources are new text and stay. On WCXB this changes 76 pages, 75 of them threads: forum F1 rises from 0.760 to 0.773 (dev) and 0.819 to 0.829 (test), and output shrinks by 30k tokens. The 31 pages that drop are threads whose ground truth keeps the quoted copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Responsive layouts render a section once for desktop and once for mobile (d-none d-md-block / d-md-none, hidden-md-down / hidden-md-up), shops render a grid and a list view, and forums print a post's body twice; both copies are visible to the Preprocessor, so the Markdown repeated them. A run of two or more consecutive blocks of 100+ characters that all repeat earlier blocks is such a copy and is dropped. A single repeated block is kept: meal plans, specs per variant and API parameters repeat one block legitimately, and the Mozilla fixtures repeat paragraphs. On WCXB 38 pages change (25 up, 10 down) and output shrinks by 10k tokens; the article corpora are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hidden element holding over 30% of the page is kept because anti-flicker snippets hide the whole page until scripts run. Such a wrapper holds the page's <h1>. A large hidden block beside a visible title is not one: a hidden raw-Markdown copy of the article (Zapier), BigCommerce's hidden custom-field list, a closed product-comparison panel. They were kept and doubled or swamped the output. Scrapinghub F1 0.9451 -> 0.9467, Mozilla 0.9415 -> 0.9460; on WCXB 4 pages rise (dev/0637 +0.54, dev/0604 +0.34, dev/4258 +0.33) and 2 fall: a collapsed "Methods" section the ground truth keeps (dev/4654), and an eBay page where removing the hidden dialog left seller feedback dominant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Legal footnotes, affiliate disclosures and "rates subject to change" notes sit inside the main region, often in the same rich-text widget as the content, and were rendered after it, sometimes as a third of the output. Named disclaimers or disclosures, they are now secondary like comment sections and author bios: they do not count against the article and article cleaning drops them. The singular "disclosure" names the <details> widget pattern and stays content, and WordPress category-* and tag-* terms on a post are not names of a block. On WCXB 35 pages change (28 up, 7 down), output shrinks by 11k tokens, and the article corpora are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum software renders each member's signature under every post (XenForo message-signature, vBulletin and phpBB signature, Invision signature), so the same lines repeat through a thread, and on threads with long signatures they outscored the posts in Readability scoring. A class or id whose last word is "signature" now names page furniture; names that only start with it, such as signature-dish or signature-collection, do not. On WCXB 39 pages change, 36 up and 3 down: forum F1 rises from 0.773 to 0.785 on dev, and dev/2201 from 0.35 to 0.87. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forum software prints a panel of member details next to every post: avatar, rank, join date, message and reaction counts, location (XenForo message-user and message-userExtras, phpBB postprofile, Invision cAuthorPane, vBulletin userinfo). Repeated through a thread it was a sixth of the output on many forums. Classes whose last words name such a panel are now page furniture; the token must end there, so XenForo's post body, message-userContent, is not one. On WCXB 48 pages change, 42 up and 3 down: forum F1 rises from 0.785 to 0.794 on dev, output shrinks by 20k tokens, and the article corpora are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Widgets that load their content with scripts leave a placeholder such as
"Loading Questions..." or "Loading reviews…" in the HTML, and only a
bare "Loading" was recognized: REI product pages ended with an empty
"Questions & Answers" heading over "Loading Questions...". One or two
words between "Loading" and an ellipsis now count too; without the
ellipsis ("Loading dock") the text stays. With the placeholder gone the
heading has no section left and is dropped as before.
On WCXB and Scrapinghub 7 pages change, all up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…page Boilerplate removes semantic chrome (nav, page header and footer, chrome roles) and blocks whose class names guess at furniture. On kress.com's mower listing the products sit in section.related-product, so the guesses left 178 characters and the page came out empty. Removals are now decided first and applied after. When they would leave under 200 characters, the largest guessed removal stays, provided it holds 200 characters itself and is not named after a place on the page (sidebar, footer, menu) or an overlay (modal, cookie banner). Script-rendered shells, whose guesses are drawers, sidebars and sign-up buttons, stay empty. On WCXB this changes 2 pages: dev/4247 0.00 -> 1.00, test/5301 0.00 -> 0.20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One example per behavior added on this branch: React streamed segments, closed accordion panels, Webflow's hidden class, large hidden copies beside the title, articles split into blocks, disclaimers, noise names with content words, forum signatures and author panels, the empty-page fallback, repeated quotes and sections, and loading placeholders. Each fails on main; a quote of text that is not on the page stays, on both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quotes are compared only up to the last quote block, word 6-grams are kept as hashes instead of joined strings, and pages without a repeated long block skip grouping runs. Output is unchanged; Markdown cleanup on 111 WCXB pages took 16 ms less. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SECONDARY is matched against every element's classes in several passes (scoring, the article's share, article cleaning), so the disclaimer alternative made extraction about 3% slower in an in-process A/B on 126 WCXB pages. Boilerplate matches its vocabulary once per distinct class list, and a disclaimer is furniture on any page, so the names move there. Quality is unchanged within noise: 13 WCXB pages change, dev F1 +0.0002, test -0.0004, the article corpora are identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 28, 2026
…page Sibling sections extend an article split into blocks. On a page whose top candidate falls short of the article share they also switched page mode to article mode, which drops every band they leave out: a landing page lost its intro, a claim and its call to action, a ranking page its introduction. The share is now judged on the top candidate and its Readability siblings. When only the sections reach it, they must hold 90% of the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
React streams last what waits on another service, on shops mostly carousels of recommended products. Placed into main, they pushed the product block of a product page below the share content selection needs, and the whole page was rendered. Segments that help (article bodies, product details, FAQ answers) have links at most a third of their text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"lg:h-[calc(100vh-var(--nav-height))]" read as "nav" furniture, so a product configurator was dropped once streamed content made it less than half of the page. The utility name around the brackets is still read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block named only as a disclaimer stays when it is the footnote to a mark
in the element just before it ("40-46 states*") or a paragraph continuing
the prose paragraph before it. A modifier such as hasDisclaimer names the
block that has a disclaimer, so it no longer counts as one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without its title block, for example once an affiliate disclosure no longer lifts the block's score, the content's first H1 can be a section heading below the opening paragraphs, and it stood as the page's only H1. Such an H1 is now demoted under the metadata title. A headline after a caption or a promo line stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing the member panel whole also removed the only name on many posts: XenForo 1, phpBB and vBulletin threads lost who wrote each reply (dev/0520 kept none of its 15 names). The panel is now reduced to one line, "**name**", with the post's date when the panel holds it; avatars, ranks, join dates and counts still go. A panel that names no one goes whole, and no line is written when the post opens with a by-line that names the poster, as phpBB's does. On WCXB against PR 16 at d7c3e6b, dev forum F1 rises from 0.7943 to 0.7955 (29 pages change, 21 up) and test forum F1 falls from 0.8559 to 0.8534, as its references mostly leave names out. The article corpora are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping a reply's quote of an earlier post left replies that answer nothing, "X posted..." lines with no quote after them, and lost a testimonial a page shows twice. A quote whose 6-grams mostly repeat earlier posts now keeps the quoter's own opening, up to the sentence that reaches 12 words (lead-ins like "Sam said:", nested quotes and links left out), then an ellipsis. A quote with fewer than 12 more words stays whole. Only posts count as the source, not earlier quotes. On WCXB against PR 16, dev F1 0.8745 -> 0.8744 and test 0.9002 -> 0.8998. Forum output keeps about 60% of the rule's token savings (dev 2141 tokens per page, 2067 when dropping, 2244 when keeping quotes whole). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A disclaimer can be kept as a note only when its name is furniture for the disclaimer word alone. The word was cut without its separators, so "footer__disclaimer" read as "footer__", which the footer pattern does not match, and the footer's fine print after a prose paragraph was kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The segment's id went into a CSS selector as is, so an id such as "S:1']" or "S:\" raised Nokolexbor::Lexbor::UnexpectedDataError and the whole conversion failed. The slots are now read once into a hash keyed by number. A slot that sat inside a fallback removed for an earlier segment is skipped, as the selector did when it no longer found it. All 339 segment ids on WCXB's 39 streamed pages are hex, so the output there does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Suspense fallback can hold boundaries of its own. The fallback was cut
at the first "/$" comment, which closes the nested boundary, so the rest
of the outer fallback ("Still loading.") stayed beside the streamed
content. The cut now counts "$", "$?" and "$!" comments in and "/$"
comments out, and stops at the one that closes the slot's boundary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pattern for an arbitrary value let "[" into its contents, so on Ruby 2.7, whose regexp engine has no memoization, a class with a run of unclosed "[" was rescanned from each one. 30,000 of them took 8.05 s against 1.0 ms on main; they now take 4.1 ms. Brackets inside the value are allowed one level deep, possessively, so a variant such as "[&:not([hidden])_.share-row]:flex" is stripped whole rather than from its inner "[" on, which left "share-row" to read as a share widget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sentence-end pattern could start inside a run of ".", "!", "?" or "…" and backtrack through it, so on Ruby 2.7 a quoted paragraph with a long run of dots was rescanned from each one: 30,000 dots took 11.98 s against 4.6 ms on main. The match now starts at the first mark of a run and its quantifiers are possessive; it takes 7.3 ms. It finds the same sentence ends as before on 200,000 random strings of marks, quotes, brackets, letters and spaces, on Ruby 2.7, 3.4 and 4.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To find where a quote's opening ends, the whole line up to each sentence end was cut out and its words counted again, so a line of many short sentences was read once per sentence, on any Ruby: 64,000 "! " took over 30 s against 17 ms on main. The head now grows a sentence at a time and only the new text is counted, once any bracket or code span in it closes, so a link that spans sentences is still read whole; it takes 73 ms. On 1,000,000 random lines it ends the opening where it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any class or id ending in "signature" was noise, which also removed a
function's signature in API docs (TypeDoc tsd-signature, Javadoc
member-signature, YARD's h3.signature heading itself) and a post whose
body is marked as having one ("post-text has-signature"): with no
content word in its name and under 200 characters, nothing saved it.
A signature now needs a post around it: a name that, without the
signature word, matches post, message, comment, reply, thread, topic,
forum, answer and the like, or an ancestor named so. "has-signature" and
"with-signature" are modifiers and never match. Of the 515 signature-named
elements on WCXB, all but 16 have that context, and those 16 are icons,
empty spans and wiki headings about PGP signatures. Only three pages
change, each keeping an artist's or a CEO's signature image; F1 is
unchanged. On Mozilla, lemonde-1 keeps its "Le Monde | date | Par"
byline (p.bloc_signature) again, as main does: 0.994 -> 0.984 precision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A large hidden element is kept as an anti-flicker wrapper only when it
holds every h1 on the page. Many themes mark up the site logo as an
<h1> in the header, outside that wrapper, so the whole page was removed
and the output was just the logo. An h1 in a nav, an image-only h1, or
one whose text is a link to the home page ("/", "/en/", the site root)
no longer counts on either side. A linked h1 that goes elsewhere still
counts.
No WCXB page changes: the h1s outside its large hidden blocks are real
titles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WooCommerce puts the product's category, tag and brand terms on the
product element ("product_cat-cookies", "product_tag-social-gifts"),
as WordPress does tag-* and category-* on a post. Those were already
ignored when deciding whether a noise name is excused by a content
word; the WooCommerce ones were not, so a product filed under
"cookies" or tagged "social gifts" was removed as a cookie banner or a
share bar, and a lesser block took its place as the content.
No WCXB page changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasir-serpapi
approved these changes
Nov 9, 2026
jasir-serpapi
added a commit
that referenced
this pull request
Nov 9, 2026
lib/nokodown.rb conflicted only on the requires: both member_panel (#16) and post_attribution are loaded. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
jasir-serpapi
added a commit
that referenced
this pull request
Nov 9, 2026
On XenForo themes that print the name in the member panel and again in the post header (dev 0523), PostAttribution read the name from the header, and #16's MemberPanel reduced the panel beside it to a second "**name**" line: every post opened with two lines, 20 pairs on 0523. The member panels inside an attributed post, outside its body, are now part of what the line replaces. Boilerplate still reduces them, so the selection does not change, and the line they leave goes with the headers when the content is selected. WCXB: only dev 0523 changes, 0.627 -> 0.632; no page has a pair left but dev 2797, where the post under the first line was pruned (as on main). Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the 2,008 WCXB pages, nokodown
main(5833a27) loses by 0.1 F1 or more to Trafilatura 2.3.1 on 130 pages and to rs-trafilatura 0.1.1 on 149. Tracing where the ground-truth text disappears on each of those pages (preprocessing, boilerplate, content selection, pruning, Markdown), plus a diagnosis of the 131 pages that lose on precision, points to a handful of common causes rather than one-off sites:<div id="S:n">elements at the end of the body, and a script moves them into place; without the script they were dropped (39 pages, e.g. fool.com's whole article body). Closed WAI-ARIA accordion panels hidden withdisplay:nonewere dropped too.related-postsorrelated products entry-summarysurvived because a content word in the name excused them. Legal disclaimers had no name at all. "Loading Questions..." placeholders stayed, and Webflow's hidden class wasn't recognized.section.related-product), the page came out empty.Change
Thirteen behavior commits, each measured on its own (table below). Then specs, the README, a speedup with byte-identical output, and one move for CPU:
div#S:nintotemplate#B:nortemplate#P:n, dropping the loading fallback up to its<!--/$-->marker when the page still has it. All 274$RC/$RSinstructions in the corpus pairS:nwithB:n/P:n.aria-controlspoints at, and that is a region or labelled by its header, is a closed panel and stays.w-condition-invisiblecounts as a hiding class.tag-*,category-*andhas-*tokens are ignored.<h1>, as an anti-flicker wrapper does.disclosure, which names<details>widgets) are removed. They first went intoSECONDARY; commit 15 moves them to page furniture.signature.postprofile,message-user,user-info/-details/-extras/-stats,poster-info,author-pane. The name must end the class token, so XenForo's post bodymessage-userContentstays.main, plus a control); the README's "What it removes" lists the new removals.SECONDARYis matched against every element in several passes, and the disclaimer names made extraction about 3% slower. Boilerplate matches its vocabulary once per distinct class list, so the names move there. Quality is the same within noise.Eight commits from the review round (see "Review round: blind judge" below) narrow rules 1, 3, 6, 9 and 11, fix two defects those rules exposed (19, 21), and correct one of the narrowings (24):
main.lg:h-[calc(100vh-var(--nav-height))]no longer reads as "nav".<p>continuing a prose paragraph.hasDisclaimernames the block that has a disclaimer, not the note.**name**, plus the post's date when the panel holds it. A panel that names no one goes whole, and no line is written when the post opens with a by-line naming the poster (phpBB).footer__disclaimerread asfooter__, which the footer pattern does not match, and a footer disclaimer right after a prose paragraph was kept. The separators now go with the word:footer__disclaimer,footer_disclaimer,site-footer-disclaimeranddisclaimer-footerstay furniture. 17 such class names sit on 18 WCXB pages; no WCXB output byte changes, and the Scrapinghub and Mozilla scores are unchanged.Results
WCXB scored with its word-level F1 and the "without" snippets (boilerplate leaked); tokens are o200k on the Markdown. Articles are scored with the Scrapinghub shingle metric.
mainF1 by page type, all 2,008 pages:
mainPer commit, each against the one before (commits 1–16; commits 17–24 are measured in the review round section: 17–23 each alone on top of commit 16 and all together, 24 against 23). CPU is an in-process A/B, alternating page by page, min of 3 runs, on every 16th WCXB page (126). The machine's load average was 200–400, so ±1.5% is noise.
mainCPU of the branch at commit 16 against
main, 502 WCXB pages, run in both orders:mainfirstAfter the review round,
benchmark/speed.rb --runs 3on the 511 WCXB test pages, one checkout at a time. The review commits cost nothing measurable:mainA first run of this branch, at 19.6 / 58.4 ms, ran while other jobs loaded the machine and is left out.
The pages that cost most are pages that now extract more content: eBay dev/0614, Product Hunt dev/2914, fool.com dev/4503. Pages that now drop a large hidden block get faster (dev/0637: 462 → 159 ms).
The pages named in the investigation:
mainaria-hiddencarousel slides; not changedelementor-widget-container−25,elementor-post__text+25) pick a teaser gridPages that got worse
180 WCXB pages rise and 65 fall (by more than 0.0005); 13 fall by 0.05 or more. At commit 16 it was 183, 57 and 17: commits 17 and 18 bring dev/0625, dev/4332 and dev/4477 back to
main, and commit 23 lifts dev/0545 to −0.02.view--hrsa-promoted-blocks, the site's own promo cards, which the ground truth keeps.Tried and dropped
All measured on the full benchmark:
sidebar-main): dev/4091 −0.91 and test/5004 −0.73, whose content wrappers are namedcontent-sidebar.<main>region root: test F1 −0.0012.Left for later
mainor a whole tab widget.widget, andmain+25.aria-hidden.Review round: blind judge
Blind pairwise judges (Claude, given the reference and both outputs, scoring completeness, noise and formatting) compared
mainwithmain+ #16 + #18 + #19 on the 50 of 210 panel pages where the two differ. The open PRs together won 29 andmain17, with 4 ties (mean overall score 6.82 against 6.03). But on the 32 pages that only this PR changes, this PR won 13 andmain17 (2 ties), although it raised WCXB F1. Every pagemainwon came from this PR alone: #18 and #19 equalmainthere.What the judges found, and the cause
mainhas 2). A quote of an earlier quote (a testimonial shown twice, nginx examples, a customer quote) was dropped as well.mainkept 13, buried in stats).hasDisclaimerread as a disclaimer.<main>was rendered. On another shop, the streamed description lowered the configurator's share and exposedlg:h-[calc(100vh-var(--nav-height))]read as "nav", so the configurator was removed.The 17 pages
mainwon, before and afterWCXB F1 against the reference.
mainmain: the second testimonial is back.main: the repeated customer quote is back.**name**line; the member stats stay out.main: the state-law note is back.main: H1, intro, claim and call to action are back.main.main: about 3,800 tokens back to 320.[itemprop~=author]would let commit 22 write "name · asked Jan 4, 2024 at 11:30"; left for a follow-up because it touches about 20 Stack Exchange pages.15 of the 17 are fixed or improved (4564 and 0576 in part); 4309 and 4017 are not.
The 23 pages this PR won
11 are byte-identical to commit 16. The other 12 change only by quote openings and name lines, plus four restored "See SUMMARY.md" links on dev/0498 (0.993 → 0.998). Of those 12, all stay above
mainon F1 except dev/0545 (main0.924, commit 16 0.867, now 0.903), which now differs frommainonly in its quote, cut to one sentence; all but 0498 stay belowmainon tokens. Four forum pages give back part of their F1 gain because their references leave the quotes out: dev/0538 0.968 → 0.896 (main0.802), 0537 0.971 → 0.934, 0547 0.486 → 0.461, 0579 0.742 → 0.720. On 0538 the replies are tied to the numbered questions again, at 5,629 tokens againstmain's 7,006.Numbers
Each fix alone on top of commit 16 (d7c3e6b), and all together:
By page type,
main/ commit 16 / now:main: dev +0.0073 (commit 16: +0.0066), test +0.0078 (commit 16: +0.0084). Test forum gives back 0.0061, because its references mostly leave poster names and quoted copies out.main: dev 78 up and 15 down (commit 16: 79 and 24), test 37 up and 2 down (commit 16: 41 and 2).Combined with #18
Merging #18 on top (scratch branch, not pushed): on dev/0520 every post has one name line. dev/0558 and dev/4309 get no attribution line from either PR (conditional cleaning removes MyBB's poster name, on
maintoo; 4309 as above). On dev/0523 every post gets #18's "name · date" line and this PR's "name" line (20 pairs). #18'sPostAttribution#markhas to remove the member panels inside each post it attributes, a 9-line change on #18's side; with it there are no duplicates on the 8 forum pages checked.Blind judge, second round
The same judges, on the same 210-page panel, compared
mainwithmain+ this PR (213130d) + #18 + #19. The outputs differ on 49 pages: the PRs won 42 andmain3, with 4 ties (mean overall score 7.43 against 5.79; the first round was 29 to 17, 6.82 against 6.03). On the 31 pages that only this PR changes, it won 26 andmain3, with 2 ties (first round: 13 to 17).mainstill wins:Review notes left open
Two reviews of commits 17–23 found one blocking defect, fixed in commit 24. The non-blocking notes:
main(dev/0324), or as an opening (dev/0496, −0.006 against commit 16).NAME_CLASSandDATE_CLASSinmember_panel.rbmatch broadly on synthetic markup: arank-namespan can be read as the name, atimezonespan as the date. On WCXB every name found is a poster.main.prose_note?keeps a<p class="disclaimer">right after an 80-character paragraph, where site fine print often sits. WCXB has 4 such notes; one is in its reference.How it was tested
bundle exec rspec: 261 examples, 0 failures on Ruby 2.7.8 with nokolexbor 0.6.4 and on Ruby 3.4.5 with 0.6.4 and 0.8.0 (commit 16's 241 examples also passed on Ruby 2.7.8 with 0.8.0). The 12 behavior specs of commits 1–13 fail onmain; the control (a quote of text not on the page stays) passes on both. Of the 21 specs added in the review round (one old quote spec replaced), 16 fail on commit 16 and 5 are guards (fine print still removed, footer-named fine print still removed after a prose paragraph, a stats-only panel still removed whole, a phpBB by-line still the only name, a headline below a photo caption kept). The footer-named guard fails at commit 23.mainwon and the 23 pages this PR won were read and compared withmainand commit 16.benchmark/speed.rbfor the review round.🤖 Generated with Claude Code