Repository navigation
Write each forum post's author and date above its body - #18
Merged
Merged
Conversation
Forum threads lost every post's author and date, because the header they
sit in is furniture that boilerplate removal drops. Read them before that
step and, once the content is selected, write them as one line above each
post's body ("**JGD** · April 2, 2024") in place of the header, the panel
around the name and repeated copies of the date.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read a date wrapped in another from the inner one, and skip member
panel dates ("Joined <time>"). Drop ad slots posted by "Advertisement"
and write no line for a name the Pruner would strip. Take a short
reply, not the signature or footer after it, as the body. Attribute an
opening post that stands apart from the replies. Check body parts by
tag name: Node#matches? searches the whole document.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A member named Ad lost the body of every post: the ad-slot rule took "Ad", "Ads" and "Advert" for ad labels. Ad slots are now told by the labels that are never names, which the boilerplate pattern shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/nokodown.rb conflicted only on the requires: both member_panel (#16) and post_attribution are loaded. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
On XenForo themes that print the name in the member panel and again in the post header (dev 0523), PostAttribution read the name from the header, and #16's MemberPanel reduced the panel beside it to a second "**name**" line: every post opened with two lines, 20 pairs on 0523. The member panels inside an attributed post, outside its body, are now part of what the line replaces. Boilerplate still reduces them, so the selection does not change, and the line they leave goes with the headers when the content is selected. WCXB: only dev 0523 changes, 0.627 -> 0.632; no page has a pair left but dev 2797, where the post under the first line was pruned (as on main). Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The cleanup of date copies took the outermost element around any copy
of the post's date that held nothing else, anywhere in the post. Inside
the body that is content: a reply that is only a date lost its body and
so its name, a date paragraph after the text went, and so did a date in
a quote ("Booked for <date>" became "Booked for").
A copy now goes only when it sits before the body, or first in it with
more of the post after it, the "April 2, 2024 1 yr" line Invision
repeats above each post. Copies in a quote stay, and the climb stops at
the body rather than taking it.
WCXB: no page changes. The 52 copies removed on WCXB (dev 0544, 0546,
0547) all open their post's body and still go.
Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A theme that puts the post's subject in the header beside the name
("## The north bridge is closed; use the east entrance") lost it: the
author panel and the header line the attribution replaces were removed
whole, heading included, or kept whole with the name and date the line
already says.
A heading in an attributed post that holds neither the name nor the
date, outside the body, quotes and member panels, is the post's
subject. It no longer counts against the header or panel being short
enough to replace, and it is moved out before they are removed. A
heading that repeats the page's h1 is the thread's title, as vBulletin
5 prints above the opening post (test 5032, 5105), and goes with the
header as before.
WCXB: no page changes. Of the 338 headings in replaced headers and
panels on WCXB, all but those two are names (XenForo h4, Invision h3)
or member ranks inside a member panel (XenForo h5).
Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A post dated only "Yesterday at 10:40", with no year, title or datetime to read a full date from, got a line with its name alone, and the header that showed the date went with it. When there is no full date, the line now shows the date as the page does, if it is short enough to be one. WCXB: no page changes; every attributed post there has a full date in its text, title or datetime. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
To find a date's post, every ancestor of every date scanned all its siblings for another of its kind that holds a date: in a thread of n posts, each of the n dates scans the n posts, and more where each post has a class of its own and the climb goes on to the next parent. A thread of 6,400 posts took 36-41 s on Ruby 2.7 and 3.4 where main takes 3 s for the whole page. The kinds of a parent's dated children are now counted once per parent and looked up from then on. PostAttribution#call, posts with a class of their own (post-N post): 1,600 posts 2.0-2.1 s -> 61-66 ms, 6,400 posts 36 s -> 305-325 ms (Ruby 2.7 and 3.4). Whole page, 6,400 posts: 41.2 s -> 3.3 s on Ruby 2.7, main 3.0 s. Posts sharing one class: 6,400 posts 6.0 s -> 5.3 s, main 2.9 s; the rest is the Pruner on 6,400 more paragraphs. WCXB: no page changes. Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
CLOSING, the class of the sibling that ends a post, matched any
"signature" or "footer" word, so a body named for having one
("post-text has-signature") was taken for the signature itself. With no
short reply before it the post got no line, and its header stayed as on
main.
A "has-" or "with-" modifier no longer counts, as in
Patterns::SIGNATURE.
WCXB: no page changes.
Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
jasir-serpapi
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A blind LLM judge comparing outputs preferred the old code on forum threads: the current code drops every post's author and date, so a reader cannot tell who wrote which reply. The header that holds them (avatar, name, rank, post count, date, permalink) is furniture that boilerplate removal and conditional cleaning drop. Examples (dev): 0547 (Affinity, Invision), 0579 (Cruise Critic, Invision), 0545 (Evernote, Invision), 0523 (XenForo), 0550 (Linus Tech Tips, Invision). The WCXB references keep each post's author and date ("JGD\nApril 2, 2024\n\n...").
Change
New
PostAttribution, run on pages whose metadata says:forumor:qa, in two steps:#call, beforeBoilerplate, so it still sees the member panels that Recover streamed and split content, and drop repeated forum text #16 removes there): a post is the lowest ancestor of a date that has a sibling of the same kind (first class) holding a date too and that names an author. An opening post that stands apart from the replies (Rick Steves'article.topicbeforesection#replies, Vanilla) is the lowest ancestor of an earlier date that names an author and is followed by the replies. Dates come from<time>anditemprop="datePublished"/"dateCreated"; a date wrapped in another date is read from the inner one, and a date labelled as the member's ("Joined", "Member since", "Registered", "Last seen", and a few German, French and Spanish equivalents) is skipped. The author is the nearest author element before the date:itemprop="author"(itsitemprop="name"),rel="author", or a class ending inusername/author/poster/creator(ipsUsername, bigusername, username-coloured, cAuthorPane_author; not author-badge). Names and dates inside a<blockquote>belong to the quoted post and are skipped. The body is the first sibling after the date, on the way up to the post, that holds content; a sibling named signature or footer (or a<footer>) is never the body: when the climb meets one, the body is the element passed over before it that holds words of its own (a short reply with no body-like class and no<p>).#write, afterContentLocator): the line goes right before the body, and the elements it replaces go: the header line with the date when it holds only a few labels and, after the date, only links and numbers ("Add the WCXB benchmark and a cross-tool comparison #1"); the panel around the name (avatar, rank, post count) when it is short and holds no prose; copies of the same date shown on their own line ("April 2, 20241 yr"). Doing this after content selection keeps the selection exactly as on main (removing headers before scoring changed it on some XenForo pages).Patterns::AD_LABEL, whichPatterns::BOILERPLATE_TEXTnow uses too (plus "ad", "ads", "advert"), so the two stay in sync;BOILERPLATE_TEXTmatches exactly what it matched before. A name the Pruner would strip as boilerplate ("Loading", "Share", "Ad") is never used, so no line is ever left without its name; such a post keeps its body and header as on main.Format:
**JGD** · April 2, 2024, one paragraph per post. The bold name reads as a speaker label and cannot be mistaken for the first words of the post; the middle dot is not a word, so it adds no noise; there is no verb ("wrote", "said"), which would add words and collide with quote headers such as "R C-R said:". The date is the one the page shows when it is a full date (with its time when shown, as on Discourse). A relative date ("1 yr", "10 hours ago") or one without a 4-digit year ("07/03/24") gives way to the full date in itstitlewhen there is one, otherwise to the machine-readabledatetime, written as2024-04-02.Threads that sign a post below its body (Stack Exchange answers, some forum themes) are left alone: a line there would read as the next post's. A per-thread vote on whether the date sits above the body decides. Q&A pages keep the attribution where they are threads: Discourse topics declared as
QAPagekeep authors in their references, while classic Stack Exchange answers, whose references drop authors, are skipped by the rule above.Speed: Nokolexbor answers
Node#matches?by searching the whole document, so body checks now compare the tag name anditemprop, and authors are found in one pass instead of per ancestor.PostAttribution#callon dev 0544 (25 posts): about 24-30 ms -> 12-16 ms, the whole page about 101-124 ms -> 79-96 ms (median of 7 runs, two measurements on a shared machine).Review round 2 (this push)
Review round 1
<time class="date" title="Reply: ...">around<time datetime>was picked, so its 9 lines had names only. Now every line has a date (**Barbara** · 2024-07-03), and the opening post, which isarticle.topicwhile replies arearticle.reply, gets its line too.· 08/26/2025 01:05 AM); the ad slot is dropped.**Linc** · March 2025, as in the reference), 1658 0.7447 -> 0.7472 (ad slot dropped). Dev forum 0.77133 -> 0.77127, test forum unchanged (0.82027).Results
WCXB, F1 main -> branch (every page type):
PostAttributionnever runs there.Regressions
11 pages lose more than 0.0005 F1, 40 gain. The losses:
Not covered (no line):
<time>oritemprop: vBulletin 3 (bare text), vBulletin 4 (span.date), MyBB (span.post_date), SMF, NodeBB (span.timeago).<meta itemprop>(the preprocessor removesmeta).itemprop=""and only a BEM class (MessageCard__user-info__name), and its first date is the thread's "last post" date, so it is not recognized as a post.Known limits from review, none seen on WCXB:
p-body-headerbeforep-body-main) with 25 or more characters of tags after the start date gets a line, repeating the first post's.With #16
git merge-tree --write-tree origin/wcxb-losses HEAD: no conflicts (README andpatterns.rbmerge cleanly).metrics_pr16.json): dev overall 0.8745 -> 0.8748, dev forum 0.7943 -> 0.7984 (+0.0041, tokens 2067 -> 2008); test overall 0.9002 -> 0.9001 (-0.0001), test forum 0.8559 -> 0.8551 (-0.0008, tokens 2441 -> 2445); every other type unchanged in both splits. Outputs are byte-identical to the previous push's merge with Recover streamed and split content, and drop repeated forum text #16.Boilerplate, where Recover streamed and split content, and drop repeated forum text #16 removes member panels, and the five example pages all show their lines. The gain is smaller because Recover streamed and split content, and drop repeated forum text #16 already removes the member panels this PR would replace, so on test only the line's cost on references without authors is left (5032 -0.023, 5105 -0.011, 5028 -0.006).**name**line instead of removing it. On dev/0523 (XenForo) this PR reads the name from the post header and leaves the panel unmarked, so each post gets both lines (20 pairs); no other WCXB page does. Whichever of the two merges second should havePostAttribution#markremove the member panels (MemberPanel.panel?) inside each post it attributes, outside the body: 9 lines, after which the 8 forum pages checked have no duplicates.How it was tested
bundle exec rspec: 244 examples, 0 failures. 16 specs inspec/nokodown/post_attribution_spec.rbon hand-written threads: the line before each body, header and author panel dropped, a full date over a relative one and the ISO fallback, the copy of the date above the body, a header that is an<aside>, a quoted author, a one-word reply, answers signed below their body, an article byline left alone, and from the reviews: an opening post outside the list of replies, a short reply followed by a signature, a member panel's join date, a date wrapped in another date, an ad slot, a member named Ad, and a name the cleanup would remove. The member-named-Ad spec fails on the previous push (a1bc7d3).BOILERPLATE_TEXTchecked against its previous form on the ad labels and near misses ("Ad", "Ads", "Advert", "Adam", "adverts", "Advertisement:", "Sponsoredcontent", ...): same result for every one.tmp/bench/eval.sh . forum-post-attribution-finalagainst main; its outputs compared with the previous push's run (forum-post-attribution-v2), andeval.sh . forum-post-attribution-final-with-16 pr16for the merge with Recover streamed and split content, and drop repeated forum text #16, compared withforum-post-attribution-v2-with-16.benchmark/quality.rbon Scrapinghub and Mozilla (F1 read with 4 decimals; the script prints 3: 0.945 and 0.941).🤖 Generated with Claude Code