Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
5dfc3a2
Flatten elements nested deeper than 64 levels
jasir-serpapi Nov 4, 2026
8d90448
Link native video and audio players to their file
jasir-serpapi Nov 4, 2026
e3563d9
Take link targets from data-href and similar attributes
jasir-serpapi Nov 4, 2026
c5f8110
Prefer lazy-loading attributes over src without a lazy class
jasir-serpapi Nov 4, 2026
e18e4a8
Keep headings and boilerplate lines when converting the whole page
jasir-serpapi Nov 4, 2026
9b708d1
Match furniture class names one at a time
jasir-serpapi Nov 4, 2026
4f7e8e2
Compute each class signature once per extraction
jasir-serpapi Nov 4, 2026
a4cb941
Find named ancestors by walking up the tree
jasir-serpapi Nov 4, 2026
b6fe5be
Skip work on plain ASCII text when collapsing and escaping
jasir-serpapi Nov 4, 2026
1f0aa19
Resolve each link and image URL once per page
jasir-serpapi Nov 4, 2026
2b825a5
Find paragraph divs from the blocks inside them
jasir-serpapi Nov 4, 2026
07d40a2
Note the faster extraction in the changelog
jasir-serpapi Nov 4, 2026
afff56c
Drop script links that whitespace or escapes disguise
jasir-serpapi Nov 9, 2026
1492f26
Escape a player's title like any other text
jasir-serpapi Nov 9, 2026
32ef8e4
Find a player's file without searching its whole subtree
jasir-serpapi Nov 9, 2026
81258da
Keep players that sit alone in a layout table cell
jasir-serpapi Nov 9, 2026
94daa56
Judge a player's container without its "media" class
jasir-serpapi Nov 9, 2026
c972c92
Skip spacer files and fragments among lazy-loading sources
jasir-serpapi Nov 9, 2026
a752a6a
Keep symbols left by a removed image when converting the whole page
jasir-serpapi Nov 9, 2026
d63425c
Find paragraphs bottom-up where a misnested link was copied
jasir-serpapi Nov 9, 2026
220f68c
Note the disguised script links in the changelog
jasir-serpapi Nov 9, 2026
95a66c5
Trim the spaces around a URL in linear time
jasir-serpapi Nov 9, 2026
5bb04a7
Keep a URL's parentheses only where Markdown pairs them
jasir-serpapi Nov 9, 2026
99ed73d
Take any file name or path from a marked lazy image
jasir-serpapi Nov 9, 2026
4f1bfb2
Cover the players that conditional cleaning keeps for holding a player
jasir-serpapi Nov 9, 2026
15359b9
Note the closed Markdown destinations in the changelog
jasir-serpapi Nov 9, 2026
9818318
Read a marked lazy image's file name without a regex
jasir-serpapi Nov 9, 2026
c7090c5
Write a link as a bare URL only when the URL holds no Markdown
jasir-serpapi Nov 9, 2026
7f494c9
Take a query or an endpoint from a marked lazy image
jasir-serpapi Nov 9, 2026
0a3dcdd
End link text at its own bracket
jasir-serpapi Nov 9, 2026
924c2c9
Encode a URL's scheme entities without locating each one
jasir-serpapi Nov 9, 2026
43cb65a
Note the closed link labels in the changelog
jasir-serpapi Nov 9, 2026
805b87c
Read code spans in link text as Markdown does
jasir-serpapi Nov 9, 2026
6fcb4cf
Percent-encode brackets and backticks in a URL
jasir-serpapi Nov 9, 2026
ea0b543
Note the encoded URL brackets and link-text code in the changelog
jasir-serpapi Nov 9, 2026
851c2f3
Encode the user info before an IPv6 host
jasir-serpapi Nov 9, 2026
573759a
Escape a link's syntax inside link text
jasir-serpapi Nov 9, 2026
9156616
Take a bare relative URL from a marked lazy image without src
jasir-serpapi Nov 9, 2026
c45521b
Read no backtick run longer than 80 as code in link text
jasir-serpapi Nov 9, 2026
a8a3388
Escape an autolink or HTML tag inside link text
jasir-serpapi Nov 9, 2026
5f54724
Note the escaped autolinks and long backtick runs in the changelog
jasir-serpapi Nov 9, 2026
d7fbeb8
Take a marked lazy image's first source when its src is empty
jasir-serpapi Nov 9, 2026
74132a8
Escape email autolinks whatever their address starts with
jasir-serpapi Nov 10, 2026
ec672ca
Escape the Markdown that raw TeX could start
jasir-serpapi Nov 10, 2026
b9a1831
Escape the alt text of emoji images
jasir-serpapi Nov 10, 2026
f06fd1f
Double a backslash at the end of a text
jasir-serpapi Nov 10, 2026
2b62701
Write code with a backtick run too long to fence as text
jasir-serpapi Nov 10, 2026
ac1d25b
Keep a space between a code span and a backtick after it
jasir-serpapi Nov 10, 2026
d23769b
Escape a line-start backtick run only where it would open a code block
jasir-serpapi Nov 10, 2026
c9c02bd
Escape the Markdown that text pieces side by side would write
jasir-serpapi Nov 10, 2026
c9fdbc2
Note the escaping of TeX, code and adjacent text in the changelog
jasir-serpapi Nov 10, 2026
ae2b64a
Keep an escaped backtick next to the code after it
jasir-serpapi Nov 10, 2026
ae9a0f2
Leave a "<" inside closed code when escaping an email opener
jasir-serpapi Nov 10, 2026
e757aa7
Track escaped backticks instead of padding code that ends with "\"
jasir-serpapi Nov 10, 2026
381521b
Note only code that ends with "\" instead of every escaped backtick
jasir-serpapi Nov 10, 2026
5f799c1
Link embeds only from a video host's own http(s) URL
jasir-serpapi Nov 10, 2026
35d8654
Restore a link target from data attributes when href is blank
jasir-serpapi Nov 10, 2026
d92d24b
Convert everything visible when main_content is false
jasir-serpapi Nov 10, 2026
a8b08a9
Keep the destination of a link made only of images
jasir-serpapi Nov 10, 2026
3e0f899
Keep the thumbnail of a preview player inside a link
jasir-serpapi Nov 10, 2026
8c02fa8
Read the real image from more lazy-loading attributes
jasir-serpapi Nov 10, 2026
1433908
Keep every heading of a page made only of headings
jasir-serpapi Nov 10, 2026
e013239
Render a single column under a header cell as a table
jasir-serpapi Nov 10, 2026
b907cb5
Keep the headline above an article body chosen inside main
jasir-serpapi Nov 10, 2026
589b660
Stop rendering once the first max_length characters are settled
jasir-serpapi Nov 10, 2026
5752089
Walk up the parents to check that a paragraph is under the root
jasir-serpapi Nov 11, 2026
430afe0
Check class names for a furniture word before the slow patterns
jasir-serpapi Nov 11, 2026
80dedc3
Walk the tree instead of asking Lexbor for every element
jasir-serpapi Nov 11, 2026
1c101dc
Walk element pointers instead of traverse when scoring
jasir-serpapi Nov 11, 2026
c80b33b
Note max_length and the later fixes in the changelog
jasir-serpapi Nov 11, 2026
3514c03
Drop the thumbnail of a hidden preview player inside a link
jasir-serpapi Nov 11, 2026
010129d
Judge lazy srcsets and dynamic image maps as the plain attributes
jasir-serpapi Nov 11, 2026
3e0791e
Read high-resolution attributes before a dynamic image map and thumbn…
jasir-serpapi Nov 11, 2026
a257cee
Escape every pipe in a table cell, after a backslash too
jasir-serpapi Nov 12, 2026
ec6d5f9
Test the element walks on formatting copies that share a key
jasir-serpapi Nov 12, 2026
60b2c3d
Note the hidden players, lazy-image order and table pipes in the chan…
jasir-serpapi Nov 12, 2026
52384f0
Escape the pipes of a table nested in a cell once
jasir-serpapi Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,63 @@
# Changelog

## Unreleased

- Elements nested more than 64 levels below `<body>` are converted as plain
text, so deeply nested markup no longer raises `SystemStackError`, also
inside a thread with its smaller stack.
- Native `<video>` and `<audio>` players become links to the file they play,
labelled with their title. Muted videos (autoplaying backgrounds) are
skipped, and a player inside a link is replaced by its thumbnail, if it
holds one and is not hidden.
- Links without `href` take their target from `data-href`, `data-url` and
the other attributes scripts keep it in.
- An image's `data-src` (and the other attributes only lazy loaders use) is
preferred over its `src`, also without a "lazy" class, so the real image
replaces a placeholder or a low-quality thumbnail. High-resolution
attributes come first, then the largest size in `data-a-dynamic-image`,
then the other loaders' attributes and a lazy `srcset`. A value that is
no image location ("false", a number, a word) never replaces `src`.
Spacer and placeholder files no longer count as a `src`.
- With `main_content: false`, everything visible is converted, so a
selection comes out whole: headings without a section, boilerplate lines
("Advertisement", "Share"), repeated headings, paragraphs and images,
quotes, rules, buttons, form text and open dialogs are kept. Only
scripts, styles, hidden elements, form inputs, closed dialogs and
punctuation-only lines are dropped.
- Links and images whose `javascript:` or `vbscript:` scheme is disguised
by tabs, line breaks or leading control characters (`java&#9;script:`),
or that a `<base>` with such a scheme would produce, are dropped like
any other script link. A link's URL can no longer end its Markdown
destination early with a stray `)`, nor its text end the label early
with a stray `]` or a trailing backslash, and write a link of its own.
Link text holds no link, autolink or HTML tag of its own either, from
raw TeX or from a backtick run too long to fence code. Raw TeX, emoji
alt text and email addresses, whatever they start with, cannot start
Markdown that reaches past them; code with a backtick run too long to
fence is written as text; a backslash at the end of a text escapes
nothing after it; code spans side by side keep their own fences; and
text side by side cannot join into a link, an image, an autolink or a
tag.
Brackets and backticks in a URL are percent-encoded, so a URL read as
text holds no link either.
- Code inside link text keeps its brackets and backslashes as they are.
- `max_length:` returns exactly the first `max_length` characters of the
Markdown, and the same `excerpt`, and stops rendering once they are
known: at most one short render and one whole render. `nil`, the
default, changes nothing; a negative or non-Integer value raises
`ArgumentError`.
- A link whose `href` is blank takes its target from `data-href` and the
like too. A link made only of images keeps its target, as
`[![alt](src)](href)`.
- An iframe embed is linked only from an http(s) URL on the video host's
own domain, and one without a title is written as `[Video](url)`.
- A page made only of headings keeps all of them, a one-column table with
a header cell or a caption stays a table, and the H1 above an article
body chosen inside `<main>` opens the content.
- A table cell escapes every pipe, so a backslash before a pipe stays.
- Extraction is faster, with the same output: about 40% on Wikipedia's
Coffee article (364 to 214 ms on Ruby 2.7) and 27% on WCXB (ms per page).

## 0.1.0 (2026-11-02)

First release.
Expand Down
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,11 @@ public API. Other classes are internal and can change in any release.
| Option | Default | |
| --- | --- | --- |
| `url` | `nil` | Page URL. Relative links and images are resolved against it (and `<base>`), and links to the page itself are reduced to their text. |
| `main_content` | `true` | Keep only the main content. `false` converts the whole page, dropping only scripts, styles, hidden elements and form controls. |
| `main_content` | `true` | Keep only the main content. `false` converts the whole page, or a selection of it, as it is: only scripts, styles, hidden elements, form inputs, closed dialogs and punctuation-only lines are dropped, and repeats, buttons, headings without a section and lines such as "Advertisement" or "Share" stay. |
| `title` | `true` | Start with the page title as a `#` heading when the content has no `<h1>`, or when its first `<h1>` follows the opening paragraphs and so heads a section. When no content is found at all (a page rendered by scripts, a bot wall), `markdown` is empty and only `result.title` has the title. |
| `images` | `true` | Include images. Emoji images are always kept as their text. |
| `links` | `true` | Keep links. `false` keeps only their text. |
| `max_length` | `nil` | Return at most this many characters of Markdown: exactly the first `max_length` characters of what `nil` gives, and the same `excerpt`. Rendering stops once the result is known, so a huge page costs less; when the start alone cannot settle it (an `<h1>` or a thumbnail further down can change it), the whole content is rendered once more. |

`mode` says how the content was chosen: `:article` when the page was narrowed
to one body of prose, `:page` when the whole main region was kept, `:full`
Expand Down Expand Up @@ -126,7 +127,10 @@ region, or most of the page's text, is never removed because of its class name.
2. **Metadata**: JSON-LD, Open Graph and other `<meta>` tags, then the
document, in Readability's order of preference.
3. **Preprocessor**: restore lazy-loaded images (`data-src`, `srcset`,
`<noscript>` copies), drop scripts, invisible elements and form controls.
`<noscript>` copies, preferred over a placeholder or thumbnail `src`) and
link targets kept in `data-href` and similar attributes, drop scripts,
invisible elements and form controls, and turn elements nested more than
64 levels deep into plain text.
4. **Boilerplate**: remove page chrome and noise widgets across the whole page.
5. **ContentLocator**: score paragraphs with Readability's algorithm. When one
candidate holds most of the page's prose the content is narrowed to it
Expand Down Expand Up @@ -154,8 +158,8 @@ region, or most of the page's text, is never removed because of its class name.
highlighter's classes, GFM tables (rowspans and colspans filled in) for data
tables while layout tables are unwrapped, definition lists, details, TeX
from MathML, superscript and subscript digits as Unicode (10⁶, H₂O) and
other scripts as 2^(n-1) and a_n. Text is escaped only where it would
otherwise change meaning.
other scripts as 2^(n-1) and a_n, and video embeds and native players as
links. Text is escaped only where it would otherwise change meaning.

## Quality

Expand Down
5 changes: 4 additions & 1 deletion lib/nokodown.rb
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,14 @@ module Nokodown
# Options:
# url: page URL, used to resolve relative links and images
# main_content: true (default) keeps only the main content; false converts
# the whole page, removing only invisible elements and scripts
# the whole page, or a selection of it, as it is: only invisible
# elements, scripts, form inputs and punctuation-only lines go
# title: prepend the page title as an H1 when the content has none;
# a page with no content gives empty Markdown, not the title
# images: include images
# links: keep links; false renders link text only
# max_length: nil (default) or the most characters of Markdown to return;
# the result is the start of what nil gives, cut there
def self.extract(html, **options)
Extractor.new(html, **options).call
end
Expand Down
15 changes: 9 additions & 6 deletions lib/nokodown/boilerplate.rb
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,11 @@ def call
@stats = Stats.new(@body)
@total = @stats.text_chars(@body)
@shielded = shielded_keys
# Card grids repeat the same class list hundreds of times.
@furniture = Hash.new { |cache, signature| cache[signature] = signature.match?(Patterns::PAGE_FURNITURE) }
# Card grids repeat the same class list hundreds of times, and class
# names repeat across lists. No furniture pattern spans a space, so a
# list is furniture when one of its names is.
@furniture_tokens = Hash.new { |cache, token| cache[token] = Patterns.furniture?(token) }
@furniture = Hash.new { |cache, signature| cache[signature] = signature.split.any? { |token| @furniture_tokens[token] } }
@disclaimer_only = Hash.new { |cache, signature| cache[signature] = disclaimer_only?(signature) }
@signature_only = Hash.new { |cache, signature| cache[signature] = signature_only?(signature) }
@notes = Set.new
Expand Down Expand Up @@ -188,12 +191,12 @@ def footnote_to?(node, previous)
# Only a signature that is furniture for its disclaimer name alone can be
# a note; "ad-disclaimer" or "footer__disclaimer" stays furniture.
def disclaimer_only?(signature)
signature.match?(Patterns::DISCLAIMER) && !signature.gsub(DISCLAIMER_WORD, " ").match?(Patterns::PAGE_FURNITURE)
signature.match?(Patterns::DISCLAIMER) && !Patterns.furniture?(signature.gsub(DISCLAIMER_WORD, " "))
end

# Furniture for its signature name alone ("signature", "member-signature").
def signature_only?(signature)
signature.match?(Patterns::SIGNATURE) && !signature.gsub(SIGNATURE_WORD, " ").match?(Patterns::PAGE_FURNITURE)
signature.match?(Patterns::SIGNATURE) && !Patterns.furniture?(signature.gsub(SIGNATURE_WORD, " "))
end

# A member's signature is named after the post ("message-signature") or
Expand Down Expand Up @@ -240,7 +243,7 @@ def prose_section?(node, signature)
return false if signature.match?(Patterns::PLACE) || signature.gsub(DRAWER, " ").match?(Patterns::NOISE)
return false if node.css("p").size < 2

@prose_blocks.merge(node["class"].to_s.split.select { |token| token.downcase.match?(Patterns::PAGE_FURNITURE) })
@prose_blocks.merge(node["class"].to_s.split.select { |token| @furniture_tokens[token.downcase] })
true
end

Expand All @@ -250,7 +253,7 @@ def prose_section?(node, signature)
# is link text, so all characters count.
def card_classes
min_chars = @stats.chars(@body) * CARD_RUN_SHARE
runs = @body.css("[class]").group_by { |node| node["class"] }
runs = Dom.elements(@body).select { |node| node.key?("class") }.group_by { |node| node["class"] }
runs.each_with_object(Set.new) do |(classes, nodes), cards|
cards << classes if card_run?(classes, nodes, min_chars)
end
Expand Down
16 changes: 14 additions & 2 deletions lib/nokodown/container_check.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ class ContainerCheck
AD_WORDS = /\A(?:ad(?:vertising|vertisement)?|pub(?:licité)?|werb(?:ung)?|广告|Реклама|Anuncio)\z/i.freeze
LOADING_WORDS = /\A(?:(?:loading|正在加载|Загрузка|chargement|cargando)(?:…|\.\.\.)?)\z/i.freeze
POST_BODY_SHARE = 0.8
PLAYER_CLASS = /media/.freeze

def initialize(node, content_length, paragraph_keys, memo = {})
@node = node
Expand Down Expand Up @@ -44,8 +45,10 @@ def majority?
@text.length > @content_length * 0.5
end

# "media" in a class is Readability's sign of social-media widgets, but on
# a container holding a player it names the player.
def weight
@weight ||= Scorer.cleaning_weight(@node)
@weight ||= Scorer.cleaning_weight(@node, ignore: (PLAYER_CLASS if player?))
end

def data_table_involved?(outer_tables)
Expand Down Expand Up @@ -83,9 +86,18 @@ def forum_post?
!post.nil? && @text.length >= post_length(post) * POST_BODY_SHARE
end

# Native players the preprocessor kept play a file, so they are content.
def player?
return @player if defined?(@player)

@player = !@node.at_css("video, audio").nil?
end

def video_embed?
return true if player?

@node.css("object, embed, iframe").any? do |embed|
embed.attribute_nodes.any? { |attribute| attribute.value.to_s.match?(Preprocessor::VIDEO_HOSTS) }
embed.attribute_nodes.any? { |attribute| Preprocessor.video_embed?(attribute.value) }
end
end

Expand Down
4 changes: 2 additions & 2 deletions lib/nokodown/content_locator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def secondary_text(root, article_nodes, stats)
key = Dom.key(node)
next if article_keys.include?(key) || Dom.thread?(node)

if Dom.signature(node).match?(Patterns::SECONDARY) && !article_ancestor_keys.include?(key)
if Patterns.secondary?(Dom.signature(node)) && !article_ancestor_keys.include?(key)
total += stats.text_chars(node)
else
stack.concat(node.element_children.to_a)
Expand All @@ -119,7 +119,7 @@ def outermost_secondary(root)
stack = root.element_children.to_a
until stack.empty?
node = stack.pop
if Dom.signature(node).match?(Patterns::SECONDARY)
if Patterns.secondary?(Dom.signature(node))
found << node
else
stack.concat(node.element_children.to_a)
Expand Down
Loading