Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
3bb0e10
Place React streamed segments before removing hidden elements
Alaa-abdulridha Oct 19, 2026
17eff93
Keep closed accordion panels hidden by style or attribute
Alaa-abdulridha Oct 19, 2026
7d66e68
Merge sibling sections that hold as much prose as the top candidate
Alaa-abdulridha Oct 19, 2026
20d22b8
Treat Webflow's w-condition-invisible class as hidden
Alaa-abdulridha Oct 19, 2026
af4ca4c
Do not let a content word excuse a noise class name
Alaa-abdulridha Oct 19, 2026
7eee6c0
Drop quotes that repeat text already in the output
Alaa-abdulridha Oct 19, 2026
926273a
Drop sections the page renders twice
Alaa-abdulridha Oct 19, 2026
f43d83b
Keep a large hidden element only when it holds the page title
Alaa-abdulridha Oct 19, 2026
7a29bad
Treat disclaimer and disclosure blocks as secondary content
Alaa-abdulridha Oct 19, 2026
d4cab9f
Remove forum signatures
Alaa-abdulridha Oct 19, 2026
5faba40
Remove the author panel beside each forum post
Alaa-abdulridha Oct 19, 2026
eddeb4f
Treat "Loading questions..." placeholders as boilerplate
Alaa-abdulridha Oct 19, 2026
c7cda3d
Keep the largest class-name guess when furniture removal empties the …
Alaa-abdulridha Oct 19, 2026
1265c3f
Add specs for streamed segments, repeated text and forum furniture
Alaa-abdulridha Oct 19, 2026
2ea2613
Document the new removals in the README
Alaa-abdulridha Oct 19, 2026
136047d
Check repeated quotes and sections only as far as they can apply
Alaa-abdulridha Oct 19, 2026
d7c3e6b
Match disclaimers once per class list, as page furniture
Alaa-abdulridha Oct 19, 2026
38f5197
Make an article from sibling sections only when they hold nearly the …
Alaa-abdulridha Oct 28, 2026
2e2e979
Leave streamed segments that are mostly links hidden
Alaa-abdulridha Oct 28, 2026
b9cee77
Read no class names into Tailwind arbitrary values
Alaa-abdulridha Oct 28, 2026
77d7f7d
Keep disclaimers that annotate the content right before them
Alaa-abdulridha Oct 28, 2026
a71e593
Put the page title above a section H1 that follows the opening prose
Alaa-abdulridha Oct 28, 2026
a3100c3
Keep the poster's name from the member panel beside each post
Alaa-abdulridha Oct 28, 2026
59ecec4
Shorten repeated quotes to their opening instead of dropping them
Alaa-abdulridha Oct 28, 2026
213130d
Keep footer-named disclaimers as furniture
Alaa-abdulridha Oct 28, 2026
3e721e5
Merge remote-tracking branch 'origin/main' into wcxb-losses
jasir-serpapi Nov 9, 2026
0551f5f
Find streamed slots by id instead of a selector built from it
jasir-serpapi Nov 9, 2026
238a073
Remove a loading fallback past the boundaries nested in it
jasir-serpapi Nov 9, 2026
d18711f
Strip Tailwind arbitrary values without rescanning unclosed brackets
jasir-serpapi Nov 9, 2026
669fae3
Find sentence ends without rescanning a run of marks
jasir-serpapi Nov 9, 2026
45bf6a4
Grow a shortened quote's head a sentence at a time
jasir-serpapi Nov 9, 2026
ccd01eb
Remove a signature only when it belongs to a forum post
jasir-serpapi Nov 9, 2026
ea2dd7a
Do not count a site logo's h1 against an anti-flicker wrapper
jasir-serpapi Nov 9, 2026
2881a03
Read no names into WooCommerce's product terms
jasir-serpapi Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public API. Other classes are internal and can change in any release.
| --- | --- | --- |
| `url` | `nil` | Page URL. Relative links and images are resolved against it (and `<base>`), and links to the page itself are reduced to their text. |
| `main_content` | `true` | Keep only the main content. `false` converts the whole page, dropping only scripts, styles, hidden elements and form controls. |
| `title` | `true` | Start with the page title as a `#` heading when the content has no `<h1>`. When no content is found at all (a page rendered by scripts, a bot wall), `markdown` is empty and only `result.title` has the title. |
| `title` | `true` | Start with the page title as a `#` heading when the content has no `<h1>`, or when its first `<h1>` follows the opening paragraphs and so heads a section. When no content is found at all (a page rendered by scripts, a bot wall), `markdown` is empty and only `result.title` has the title. |
| `images` | `true` | Include images. Emoji images are always kept as their text. |
| `links` | `true` | Keep links. `false` keeps only their text. |

Expand All @@ -80,14 +80,23 @@ with `main_content: false`, `:none` for a document without a body.
tables of contents, found through HTML semantics (`nav`, page-level `header`
and `footer`, ARIA roles, schema.org types) and class/id names.
- Cookie banners, ads, newsletter and sign-in prompts, share bars, modals,
chat widgets, "related", "recommended" and "most popular" boxes, and, next
to an article, its comment thread.
chat widgets, "related", "recommended" and "most popular" boxes, legal
disclaimers and disclosures (but not a note continuing the paragraph before
it, or the footnote to a mark just above), and, next to an article, its
comment thread.
- On forums, the member panel and signature repeated beside every post. The
member panel leaves one line naming the poster, with the post's date when
the panel holds it. A quote that repeats a post already in the output is cut
to its opening sentence, so the reply still shows what it answers.
- On product pages, customer reviews and Q&A widgets, with their review
summaries and rating histograms. An inline rating badge ("212 reviews" next
to the price), testimonials and editorial reviews stay.
- Invisible elements (`hidden`, `aria-hidden`, inline `display: none`,
screen-reader-only and other hiding classes), keeping collapsed accordions,
tab panels and FAQ answers.
tab panels and FAQ answers. Content that React streams into hidden
containers at the end of the page is put back where the page shows it,
unless it is mostly links, as carousels of recommended products are.
- Sections rendered twice, such as a desktop and a mobile copy.
- Tracking pixels, spacers, icons, placeholders, README badges, click-tracking
URL parameters (`utm_*`, `gclid`, `fbclid`, ...), permalink anchors,
"Read more" links, code line numbers, and repeated responsive copies of the
Expand Down
1 change: 1 addition & 0 deletions lib/nokodown.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
require_relative "nokodown/url_resolver"
require_relative "nokodown/metadata"
require_relative "nokodown/preprocessor"
require_relative "nokodown/member_panel"
require_relative "nokodown/boilerplate"
require_relative "nokodown/product_reviews"
require_relative "nokodown/scorer"
Expand Down
108 changes: 93 additions & 15 deletions lib/nokodown/boilerplate.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,18 @@ class Boilerplate
PHRASING = Set.new(%w[a abbr b bdi bdo cite code data dfn em i kbd mark q s samp small span strong sub sup time u var]).freeze
AD_ATTRIBUTES = %w[data-ad-slot data-ad-unit data-adunit data-ad-client data-google-query-id data-ad-format].freeze
DRAWER = /(?<![a-z])drawer(?![a-z])/.freeze
# Terms WordPress (tag-*, category-*) and WooCommerce (product_cat-*,
# product_tag-*, product_brand-*) put on a post or product.
TAXONOMY = /(?<![\w-])(?:tag|category|has|product_(?:cat|tag|brand))-[\w-]+/.freeze
MIN_CARD_RUN = 3
MIN_LEFT = 200
CARD_RUN_SHARE = 0.2
# A note continues the prose only after a sentence or more of it, not after a line of links.
NOTE_PROSE = 80
FOOTNOTE_MARK = /\A[*†‡§]+/.freeze
# The word goes with its separators: "footer__disclaimer" without it is "footer", not "footer__".
DISCLAIMER_WORD = /[-_]*#{Patterns::DISCLAIMER.source}[-_]*/.freeze
SIGNATURE_WORD = /[-_]*#{Patterns::SIGNATURE.source}[-_]*/.freeze

def initialize(body)
@body = body
Expand All @@ -27,10 +37,18 @@ def call
@shielded = shielded_keys
# Card grids repeat the same class list hundreds of times.
@furniture = Hash.new { |cache, signature| cache[signature] = signature.match?(Patterns::PAGE_FURNITURE) }
@disclaimer_only = Hash.new { |cache, signature| cache[signature] = disclaimer_only?(signature) }
@signature_only = Hash.new { |cache, signature| cache[signature] = signature_only?(signature) }
@notes = Set.new
@prose_blocks = Set.new
@card_classes = card_classes
@card_bases = @card_classes.map { |classes| base_classes(classes) }
walk(@body)
removals = []
walk(@body, removals)
removals.delete(content_guess(removals)) if almost_empty_without?(removals)
removals.each { |node, kind| kind == :panel ? MemberPanel.reduce(node) : node.remove }
# The lines panels leave are new elements, and later passes measure by key.
Dom.assign_keys(@body) if removals.any? { |_, kind| kind == :panel }
end

private
Expand All @@ -43,29 +61,39 @@ def shielded_keys
end

# Code is never furniture, whatever classes its highlighter gives tokens.
def walk(node)
def walk(node, removals)
node.element_children.each do |child|
next if Dom::CODE.include?(child.name)

if remove?(child)
child.remove
else
walk(child)
end
kind = removal(child)
kind ? removals << [child, kind] : walk(child, removals)
end
end

def almost_empty_without?(removals)
@stats.chars(@body) - removals.sum { |node, _| @stats.chars(node) } < MIN_LEFT
end

def content_guess(removals)
guess = removals.select { |_, kind| kind == :guess }.max_by { |node, _| @stats.chars(node) }
return unless guess && @stats.chars(guess.first) >= MIN_LEFT

signature = Dom.signature(guess.first)
guess unless signature.match?(Patterns::PLACE) || signature.match?(Patterns::OVERLAY)
end

# Semantic chrome (nav, a page-level footer) goes even when it outweighs
# the content, as legal footnotes often do; class-name guesses do not.
def remove?(node)
return false if @shielded.include?(Dom.key(node))
def removal(node)
return if @shielded.include?(Dom.key(node))
# Side notes inside an article are content whatever their class says.
return false if node.name == "aside" && !aside_noise?(node)
return true if chrome_element?(node) || chrome_role?(node)
return false if dominant?(node)
return false unless advertising?(node) || chrome_label?(node) || class_noise?(node)
return if node.name == "aside" && !aside_noise?(node)
return :chrome if chrome_element?(node) || chrome_role?(node)
return if dominant?(node)
return unless advertising?(node) || chrome_label?(node) || class_noise?(node)
return if inside_sentence?(node)

!inside_sentence?(node)
MemberPanel.panel?(node) ? :panel : :guess
end

# A link or span in the middle of running text is part of that text,
Expand Down Expand Up @@ -124,7 +152,55 @@ def chrome_label?(node)
def class_noise?(node)
return false if !@prose_blocks.empty? && prose_part?(node)

furniture_class?(node) && (@card_bases.empty? || !card?(node))
furniture_class?(node) && !content_note?(node) && (@card_bases.empty? || !card?(node))
end

# Fine print about the site is furniture, but a note on the content right
# before it belongs to that content: a paragraph continuing the prose, or
# the footnote to a mark in it ("40-46 states*"). Widgets nest the note's
# text in an element named like the note.
def content_note?(node)
return false unless @disclaimer_only[Dom.signature(node)]

note = @notes.include?(Dom.key(node.parent)) || annotates_previous?(node)
@notes << Dom.key(node) if note
note
end

def annotates_previous?(node)
previous = node.previous_element
return false unless previous

prose_note?(node, previous) || footnote_to?(node, previous)
end

def prose_note?(node, previous)
return false unless node.name == "p" && previous.name == "p"

@stats.text_chars(previous) >= NOTE_PROSE && !@furniture[Dom.signature(previous)]
end

def footnote_to?(node, previous)
marks = [node, *node.css("p, li")].map { |line| Dom.text(line)[FOOTNOTE_MARK] }.compact.uniq
marks.any? { |mark| previous.content.match?(/[[:alnum:]]#{Regexp.escape(mark)}/) }
end

# Only a signature that is furniture for its disclaimer name alone can be
# a note; "ad-disclaimer" or "footer__disclaimer" stays furniture.
def disclaimer_only?(signature)
signature.match?(Patterns::DISCLAIMER) && !signature.gsub(DISCLAIMER_WORD, " ").match?(Patterns::PAGE_FURNITURE)
end

# Furniture for its signature name alone ("signature", "member-signature").
def signature_only?(signature)
signature.match?(Patterns::SIGNATURE) && !signature.gsub(SIGNATURE_WORD, " ").match?(Patterns::PAGE_FURNITURE)
end

# A member's signature is named after the post ("message-signature") or
# sits in one; a function's signature in API docs does neither.
def forum_signature?(node, signature)
signature.gsub(SIGNATURE_WORD, " ").match?(Patterns::POST) ||
node.ancestors.any? { |ancestor| ancestor.element? && Dom.signature(ancestor).match?(Patterns::POST) }
end

def card?(node)
Expand All @@ -148,7 +224,9 @@ def furniture_class?(node)
return site_header?(node) if !chrome && signature.match?(Patterns::HEADER)
return @stats.link_density(node) > 0.5 if !chrome && signature.match?(Patterns::MENU)
return false unless chrome
return false if @signature_only[signature] && !forum_signature?(node, signature)
return false if prose_section?(node, signature)
return true if signature.gsub(TAXONOMY, " ").match?(Patterns::NOISE)

!(signature.match?(Patterns::POSITIVE) && @stats.text_chars(node) >= 200)
end
Expand Down
19 changes: 16 additions & 3 deletions lib/nokodown/content_locator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ class ContentLocator
REQUIRED_SHARE = { article: 0.15, product: 0.9 }.freeze
DEFAULT_SHARE = 0.7

# Sibling sections extend an article whose top candidate holds the share.
# When only they reach it, they must hold nearly the whole page: page mode
# would keep every band, and article mode drops the bands they leave out.
WHOLE_PAGE_SHARE = 0.9

# Page types whose replies are content: forum threads and Q&A pages.
DISCUSSIONS = %i[forum qa].freeze

Expand Down Expand Up @@ -58,13 +63,21 @@ def main_region(stats)
end

def article?(result, root, stats)
article_text = result.nodes.sum { |node| stats.text_chars(node) }
required = REQUIRED_SHARE.fetch(@page_type, DEFAULT_SHARE)
return true if holds_share?(result.core, root, stats, required)
return false if result.nodes.size == result.core.size

holds_share?(result.nodes, root, stats, [required, WHOLE_PAGE_SHARE].max)
end

def holds_share?(nodes, root, stats, required)
article_text = nodes.sum { |node| stats.text_chars(node) }
return false if article_text < MIN_ARTICLE_CHARS

page_text = stats.text_chars(root) - secondary_text(root, result.nodes, stats)
page_text = stats.text_chars(root) - secondary_text(root, nodes, stats)
return true if page_text <= 0

article_text / page_text >= REQUIRED_SHARE.fetch(@page_type, DEFAULT_SHARE)
article_text / page_text >= required
end

# Comment threads and "most popular" rails beside an article do not count
Expand Down
5 changes: 5 additions & 0 deletions lib/nokodown/dom.rb
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ module Dom
KEY = "data-nokodown-key"

UUID = /\h{8}-\h{4}-\h{4}-\h{4}-\h{12}/.freeze
# One level of brackets may nest ("[&>[data-slot=icon]]"). Possessive, so
# a run of unclosed "[" is not rescanned from each one.
ARBITRARY_VALUE = /\[(?:[^\s\[\]]|\[[^\s\[\]]*+\])*+\]/.freeze
CSS_MODULE_HASH = /__(?=[\w-]{0,4}[A-Z\d])[\w-]{5}(?![\w-])/.freeze

# On a forum, the replies sit in containers named like a comment section.
Expand Down Expand Up @@ -124,6 +127,8 @@ def name_id(node)
end

def signature_of(value)
# A Tailwind arbitrary value ("lg:h-[calc(100vh-var(--nav-height))]") is CSS, not a name.
value = value.gsub(ARBITRARY_VALUE, "") if value.include?("[")
# Generated ids (Webflow "w-node-<uuid>") would otherwise read as words.
value = value.gsub(UUID, " ") if value.match?(UUID)
value = value.gsub(CSS_MODULE_HASH, "") if value.include?("__")
Expand Down
21 changes: 17 additions & 4 deletions lib/nokodown/extractor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
module Nokodown
class Extractor
TITLE_SIMILARITY = 0.75
OPENING_PROSE = 150
MIN_ARTICLE_LENGTH = 500
CONDITIONAL_SHARE = 0.5
REMOVED_LINK_SHARE = 0.5
Expand Down Expand Up @@ -90,21 +91,33 @@ def almost_nothing_kept?(copies, nodes)
kept < MIN_ARTICLE_LENGTH && kept < nodes.sum { |node| Dom.text(node).length } * 0.25
end

# One H1 at the top: the page's own if the content has one, otherwise the
# metadata title, replacing a near-duplicate heading right below it.
# One H1 at the top: the page's own if the content opens with one,
# otherwise the metadata title, replacing a near-duplicate heading right
# below it.
# Nothing found stays empty, so a script-rendered shell or a bot wall does
# not pass for a page; Result#title still has the title.
def with_title(blocks, title)
return blocks if blocks.empty?

first_h1 = blocks.find { |block| block.kind == :heading && block.level == 1 }
return demote_h1s(blocks, first_h1) if first_h1
return demote_h1s(blocks, first_h1) if first_h1 && (title.nil? || page_heading?(blocks, first_h1, title))
return blocks if title.nil?

duplicate = blocks.first(3).find do |block|
block.kind == :heading && Dom.similarity(title, Markdown::Cleanup.heading_text(block)) > TITLE_SIMILARITY
end
[Markdown::Block.new(:heading, "# #{Markdown::Text.escape(title)}", 1)] + blocks.reject { |block| block.equal?(duplicate) }
rest = demote_h1s(blocks.reject { |block| block.equal?(duplicate) }, nil)
[Markdown::Block.new(:heading, "# #{Markdown::Text.escape(title)}", 1)] + rest
end

# When the page's title block is left out of the content, the first H1
# left can be a section heading below the body's opening paragraphs. A
# headline may follow a caption or a promo line, but not the body.
def page_heading?(blocks, h1, title)
return true if Dom.similarity(title, Markdown::Cleanup.heading_text(h1)) > TITLE_SIMILARITY

opening = blocks.take_while { |block| !block.equal?(h1) }.select { |block| block.kind == :paragraph }
opening.sum { |block| Markdown::Text.plain(block.text).length } < OPENING_PROSE
end

def demote_h1s(blocks, keep)
Expand Down
Loading