Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@ jobs:
fail-fast: false
matrix:
include:
# SerpApi runs Ruby 2.7.8 with nokolexbor 0.6.4.
# The oldest supported Ruby, with the last nokolexbor that runs on it.
- ruby: "2.7"
nokolexbor: "0.6.4"
- ruby: "3.4"
nokolexbor: "0.6.4"
- ruby: "3.4"
nokolexbor: ""
- ruby: "4.0"
nokolexbor: ""
env:
NOKOLEXBOR_VERSION: ${{ matrix.nokolexbor }}
steps:
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Changelog

## 0.1.0 (2026-11-02)

First release.

- `Nokodown.extract` returns the main content of a web page as Markdown,
with Readability-style metadata (title, byline, excerpt, site name,
published time, language).
- `Nokodown.to_markdown` returns only the Markdown.
- Works on articles and on home, listing, product, documentation, forum and
service pages.
2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ source "https://rubygems.org"

gemspec

# CI pins the version SerpApi runs in production (NOKOLEXBOR_VERSION=0.6.4).
# CI also tests nokolexbor 0.6.4, the last release for Ruby 2.7 (NOKOLEXBOR_VERSION=0.6.4).
nokolexbor_version = ENV["NOKOLEXBOR_VERSION"].to_s
gem "nokolexbor", nokolexbor_version unless nokolexbor_version.empty?

Expand Down
6 changes: 5 additions & 1 deletion NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@ Copyright 2026 SerpApi

The content scoring, conditional cleaning, title and metadata heuristics are
adapted from Readability.js (https://github.com/mozilla/readability),
Copyright (c) 2010 Arc90 Inc, licensed under the Apache License, Version 2.0.
licensed under the Apache License, Version 2.0. Its NOTICE reads:

Readability
Copyright (c) 2010 Arc90 Inc
Copyright (c) 2010-2026 Mozilla and Contributors
63 changes: 37 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
<p align="center">
<a href="https://serpapi.com">
<img src="https://serpapi.com/assets/media_kit/logo-with-wordmark-gradient.svg" alt="SerpApi" width="220">
</a>
</p>

# nokodown

Any web page to main-content Markdown, at Nokolexbor speed.
Expand All @@ -8,16 +14,28 @@ language). It works on articles and on everything else: home pages, search
results, product and pricing pages, documentation, forums.

Built on [Nokolexbor](https://github.com/serpapi/nokolexbor), for the
`include_markdown` option of SerpApi's Extract API. The name is Nokolexbor's
"noko" plus Markdown's "down".
`include_markdown` option of SerpApi's [Extract API][extract]. The name is
Nokolexbor's "noko" plus Markdown's "down".

> **Don't want to fetch the pages yourself?** SerpApi's [Extract API][extract]
> takes a URL and returns the page's Markdown as JSON, along with its meta
> tags, structured data, links and images.
> [Try the Extract API][extract].

## Installation

```ruby
gem "nokodown", git: "git@github.com:serpapi/nokodown.git"
gem "nokodown"
```

Ruby 2.7 or newer, nokolexbor 0.6 or newer.
Ruby 2.7 or newer, nokolexbor 0.6 or a later 0.x release. nokolexbor 0.7 and
later need Ruby 3.1. Bundler picks 0.6.4 on older Rubies, but `gem install`
does not, so install that version first:

```sh
gem install nokolexbor -v 0.6.4
gem install nokodown
```

## Usage

Expand All @@ -41,6 +59,9 @@ Nokodown.to_markdown(html, url: "https://example.com/post") # just the Markdown
`html` is a string in any encoding. Binary strings are decoded using their
`<meta charset>`, and invalid bytes are dropped.

`Nokodown.extract`, `Nokodown.to_markdown` and `Nokodown::Result` are the
public API. Other classes are internal and can change in any release.

| Option | Default | |
| --- | --- | --- |
| `url` | `nil` | Page URL. Relative links and images are resolved against it (and `<base>`), and links to the page itself are reduced to their text. |
Expand Down Expand Up @@ -129,7 +150,9 @@ region, or most of the page's text, is never removed because of its class name.

[WCXB][wcxb] has 2,008 human-reviewed pages: 1,050 articles and 958 product,
collection, listing, forum, documentation and service (SaaS, pricing) pages,
split into a development set (1,497) and a held-out test set (511). Scores use
split into a development set (1,497) and a test set (511). The test set was
not held out: some of its pages were studied while tuning nokodown, and 139
of them have the same HTML as a development page. Scores use
its metric: word-level F1 against the reference main content, plus the share
of "must include" snippets found and of boilerplate snippets leaked. All
systems ran on the same pages with links, images and tables on; their Markdown
Expand All @@ -146,7 +169,7 @@ point.
| rs-trafilatura 0.1.1 | 0.838 | 0.873 | 12.7% | 87.7% | 2,664 | 18% |
| Trafilatura 2.3.1 | 0.816 | 0.858 | 8.1% | 81.8% | 2,552 | 17% |
| Readability.js 0.6 + Turndown | 0.770 | 0.830 | 10.0% | 76.5% | 2,899 | 23% |
| Current Extract rule | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% |
| Main-region baseline | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% |

F1 by page type on the test set (257 articles; 28 to 59 pages of each other type):

Expand All @@ -156,12 +179,12 @@ F1 by page type on the test set (257 articles; 28 to 59 pages of each other type
| rs-trafilatura 0.1.1 | 0.942 | 0.740 | 0.732 | 0.714 | 0.829 | 0.953 | 0.801 |
| Trafilatura 2.3.1 | 0.934 | 0.719 | 0.589 | 0.757 | 0.786 | 0.941 | 0.817 |
| Readability.js 0.6 + Turndown | 0.931 | 0.634 | 0.614 | 0.637 | 0.711 | 0.882 | 0.805 |
| Current Extract rule | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 |
| Main-region baseline | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 |

"Current Extract rule" is the Extract API's present `include_markdown`
behaviour (the page's `<main>`, else its only `<article>`, else `<body>`)
reproduced with nokodown's converter. See
[benchmark/compare](benchmark/compare/README.md) to reproduce the table.
"Main-region baseline" converts the page's `<main>`, else its only
`<article>`, else `<body>`, with nokodown's converter and no main-content
detection. See [benchmark/compare](benchmark/compare/README.md) to reproduce
the table.

```sh
bundle exec ruby benchmark/wcxb.rb tmp/corpora/wcxb test
Expand Down Expand Up @@ -202,34 +225,22 @@ bundle exec rake benchmark:speed
ruby benchmark/speed.rb DIRECTORY --per-page # any folder of .html or .html.gz files
```

## Using it in the Extract API

The Extract API currently fills `text_content.markdown` from the page's
`<main>`, its only `<article>`, or the whole body. With this gem:

```ruby
markdown = Nokodown.to_markdown(html, url: final_url)
```

`main_content: false` gives the whole-page conversion if a request needs it.
Truncation to 50,000 characters stays in the engine, and `include_tags` /
`exclude_tags` can be applied to the HTML before it is passed in.

## Development

```sh
bundle install
bundle exec rspec
NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the version SerpApi runs
NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the last nokolexbor for Ruby 2.7
```

Specs use RSpec's `should` syntax, like the main SerpApi app.
Specs use RSpec's `should` syntax.

## License

Apache License 2.0. The scoring and cleaning heuristics are adapted from
[Readability.js][mozilla]; see [NOTICE](NOTICE).

[benchmark]: https://github.com/scrapinghub/article-extraction-benchmark
[extract]: https://serpapi.com/extract-api
[mozilla]: https://github.com/mozilla/readability
[wcxb]: https://github.com/Murrough-Foley/web-content-extraction-benchmark
10 changes: 5 additions & 5 deletions benchmark/compare/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ bundle exec rake benchmark:corpora # WCXB into tmp/corpora/wcxb
W=tmp/corpora/wcxb R=tmp/runs

ruby benchmark/compare/run_nokodown.rb $W $R/nokodown
ruby benchmark/compare/run_nokodown.rb $W $R/current-extract current
ruby benchmark/compare/run_nokodown.rb $W $R/main-region baseline

python3 -m venv tmp/venv && tmp/venv/bin/pip install -r benchmark/compare/requirements.txt
tmp/venv/bin/python benchmark/compare/run_python.py trafilatura $W $R/trafilatura
Expand All @@ -19,15 +19,15 @@ tmp/venv/bin/python benchmark/compare/run_python.py rs-trafilatura $W $R/rs-traf
for i in 0 1 2 3; do node benchmark/compare/run_readability.js $W $R/readability $i 4 & done; wait

tmp/venv/bin/python benchmark/compare/metrics.py $W $R tmp/metrics.json \
nokodown rs-trafilatura trafilatura readability current-extract
nokodown rs-trafilatura trafilatura readability main-region
```

rs-trafilatura has no macOS wheel; building it needs a Rust toolchain, and on
Python 3.14 also `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1`.

`current-extract` reproduces the Extract API's current rule (the page's
`<main>`, else its only `<article>`, else `<body>`) using nokodown's converter,
so it isolates the effect of main-content detection.
`main-region` is the baseline: the page's `<main>`, else its only `<article>`,
else `<body>`, converted with nokodown's converter, so it isolates the effect
of main-content detection.

Besides the benchmark's F1 and snippet rates, `metrics.py` reports what an LLM
pays for: tokens per page (o200k tokenizer), the share of tokens spent on URLs
Expand Down
14 changes: 7 additions & 7 deletions benchmark/compare/run_nokodown.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@

# Writes nokodown's Markdown for every WCXB page: OUT/<split>/md/<id>.md.
#
# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [current]
# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [baseline]
#
# With "current", it reproduces the Extract API's current rule instead: the
# page's <main>, else its only <article>, else <body>, converted without
# main-content detection.
# With "baseline", it converts the main-region baseline instead: the page's
# <main>, else its only <article>, else <body>, without main-content
# detection.

require "fileutils"
require "json"
require "zlib"
require_relative "../../lib/nokodown"

def current_rule(html, url)
def main_region(html, url)
doc = Nokolexbor::HTML(Nokodown::Charset.utf8(html))
articles = doc.css("article")
region = doc.at_css("main") || (articles.size == 1 ? articles.first : nil) || doc.at_css("body")
Expand All @@ -23,15 +23,15 @@ def current_rule(html, url)
end

wcxb, out, mode = ARGV
abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [current]" unless wcxb && out
abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [baseline]" unless wcxb && out

%w[dev test].each do |split|
FileUtils.mkdir_p(File.join(out, split, "md"))
Dir[File.join(wcxb, split, "ground-truth", "*.json")].sort.each do |path|
id = File.basename(path, ".json")
url = JSON.parse(File.read(path))["url"]
html = Zlib::GzipReader.open(File.join(wcxb, split, "html", "#{id}.html.gz"), &:read)
markdown = mode == "current" ? current_rule(html, url) : Nokodown.to_markdown(html, url: url)
markdown = mode == "baseline" ? main_region(html, url) : Nokodown.to_markdown(html, url: url)
File.write(File.join(out, split, "md", "#{id}.md"), markdown)
end
end
2 changes: 1 addition & 1 deletion lib/nokodown/metadata.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require "cgi"
require "cgi/escape"
require "json"

module Nokodown
Expand Down
4 changes: 3 additions & 1 deletion lib/nokodown/url_resolver.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ module Nokodown
# tracking parameters, click redirectors and image resize options cost
# tokens and tell a reader nothing.
class UrlResolver
# From Ruby 3.4 URI::DEFAULT_PARSER is the RFC 3986 parser, whose unescape is obsolete.
URI_PARSER = defined?(URI::RFC2396_PARSER) ? URI::RFC2396_PARSER : URI::DEFAULT_PARSER
TRACKING_PARAM = /\A(?:utm_[a-z_]+|fbclid|gclid|gclsrc|dclid|gbraid|wbraid|msclkid|yclid|igshid|mc_cid|mc_eid|_hsenc|_hsmi|mkt_tok|__s|
_gl|_ga|srsltid|trk|icid|cmpid|hsCtaTracking|ref_|pd_rd_[a-z]+|pf_rd_[a-z]+)\z/ix.freeze
# Amazon session, search-rank and affiliate parameters; on other sites
Expand Down Expand Up @@ -223,7 +225,7 @@ def decode(value)
end

def unescape(value)
decoded = URI::DEFAULT_PARSER.unescape(value)
decoded = URI_PARSER.unescape(value)
decoded if decoded.valid_encoding?
end

Expand Down
12 changes: 10 additions & 2 deletions nokodown.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,16 @@ Gem::Specification.new do |spec|
spec.license = "Apache-2.0"
spec.required_ruby_version = ">= 2.7"

spec.files = Dir["lib/**/*.rb", "LICENSE", "NOTICE", "README.md"]
spec.files = Dir["lib/**/*.rb", "CHANGELOG.md", "LICENSE", "NOTICE", "README.md"]
spec.require_paths = ["lib"]

spec.add_dependency "nokolexbor", ">= 0.6"
spec.metadata = {
"homepage_uri" => spec.homepage,
"source_code_uri" => spec.homepage,
"changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md",
"bug_tracker_uri" => "#{spec.homepage}/issues",
"rubygems_mfa_required" => "true"
}

spec.add_dependency "nokolexbor", "~> 0.6"
end