diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3b72486..e1462ce 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -13,13 +13,15 @@ jobs:
fail-fast: false
matrix:
include:
- # SerpApi runs Ruby 2.7.8 with nokolexbor 0.6.4.
+ # The oldest supported Ruby, with the last nokolexbor that runs on it.
- ruby: "2.7"
nokolexbor: "0.6.4"
- ruby: "3.4"
nokolexbor: "0.6.4"
- ruby: "3.4"
nokolexbor: ""
+ - ruby: "4.0"
+ nokolexbor: ""
env:
NOKOLEXBOR_VERSION: ${{ matrix.nokolexbor }}
steps:
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..0325dfc
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 (2026-11-02)
+
+First release.
+
+- `Nokodown.extract` returns the main content of a web page as Markdown,
+ with Readability-style metadata (title, byline, excerpt, site name,
+ published time, language).
+- `Nokodown.to_markdown` returns only the Markdown.
+- Works on articles and on home, listing, product, documentation, forum and
+ service pages.
diff --git a/Gemfile b/Gemfile
index f258dbb..dd733a3 100644
--- a/Gemfile
+++ b/Gemfile
@@ -4,7 +4,7 @@ source "https://rubygems.org"
gemspec
-# CI pins the version SerpApi runs in production (NOKOLEXBOR_VERSION=0.6.4).
+# CI also tests nokolexbor 0.6.4, the last release for Ruby 2.7 (NOKOLEXBOR_VERSION=0.6.4).
nokolexbor_version = ENV["NOKOLEXBOR_VERSION"].to_s
gem "nokolexbor", nokolexbor_version unless nokolexbor_version.empty?
diff --git a/NOTICE b/NOTICE
index d235277..3506969 100644
--- a/NOTICE
+++ b/NOTICE
@@ -3,4 +3,8 @@ Copyright 2026 SerpApi
The content scoring, conditional cleaning, title and metadata heuristics are
adapted from Readability.js (https://github.com/mozilla/readability),
-Copyright (c) 2010 Arc90 Inc, licensed under the Apache License, Version 2.0.
+licensed under the Apache License, Version 2.0. Its NOTICE reads:
+
+ Readability
+ Copyright (c) 2010 Arc90 Inc
+ Copyright (c) 2010-2026 Mozilla and Contributors
diff --git a/README.md b/README.md
index e21f99e..9290a45 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,9 @@
+
+
+
+
+
+
# nokodown
Any web page to main-content Markdown, at Nokolexbor speed.
@@ -8,16 +14,28 @@ language). It works on articles and on everything else: home pages, search
results, product and pricing pages, documentation, forums.
Built on [Nokolexbor](https://github.com/serpapi/nokolexbor), for the
-`include_markdown` option of SerpApi's Extract API. The name is Nokolexbor's
-"noko" plus Markdown's "down".
+`include_markdown` option of SerpApi's [Extract API][extract]. The name is
+Nokolexbor's "noko" plus Markdown's "down".
+
+> **Don't want to fetch the pages yourself?** SerpApi's [Extract API][extract]
+> takes a URL and returns the page's Markdown as JSON, along with its meta
+> tags, structured data, links and images.
+> [Try the Extract API][extract].
## Installation
```ruby
-gem "nokodown", git: "git@github.com:serpapi/nokodown.git"
+gem "nokodown"
```
-Ruby 2.7 or newer, nokolexbor 0.6 or newer.
+Ruby 2.7 or newer, nokolexbor 0.6 or a later 0.x release. nokolexbor 0.7 and
+later need Ruby 3.1. Bundler picks 0.6.4 on older Rubies, but `gem install`
+does not, so install that version first:
+
+```sh
+gem install nokolexbor -v 0.6.4
+gem install nokodown
+```
## Usage
@@ -41,6 +59,9 @@ Nokodown.to_markdown(html, url: "https://example.com/post") # just the Markdown
`html` is a string in any encoding. Binary strings are decoded using their
``, and invalid bytes are dropped.
+`Nokodown.extract`, `Nokodown.to_markdown` and `Nokodown::Result` are the
+public API. Other classes are internal and can change in any release.
+
| Option | Default | |
| --- | --- | --- |
| `url` | `nil` | Page URL. Relative links and images are resolved against it (and ``), and links to the page itself are reduced to their text. |
@@ -129,7 +150,9 @@ region, or most of the page's text, is never removed because of its class name.
[WCXB][wcxb] has 2,008 human-reviewed pages: 1,050 articles and 958 product,
collection, listing, forum, documentation and service (SaaS, pricing) pages,
-split into a development set (1,497) and a held-out test set (511). Scores use
+split into a development set (1,497) and a test set (511). The test set was
+not held out: some of its pages were studied while tuning nokodown, and 139
+of them have the same HTML as a development page. Scores use
its metric: word-level F1 against the reference main content, plus the share
of "must include" snippets found and of boilerplate snippets leaked. All
systems ran on the same pages with links, images and tables on; their Markdown
@@ -146,7 +169,7 @@ point.
| rs-trafilatura 0.1.1 | 0.838 | 0.873 | 12.7% | 87.7% | 2,664 | 18% |
| Trafilatura 2.3.1 | 0.816 | 0.858 | 8.1% | 81.8% | 2,552 | 17% |
| Readability.js 0.6 + Turndown | 0.770 | 0.830 | 10.0% | 76.5% | 2,899 | 23% |
-| Current Extract rule | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% |
+| Main-region baseline | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% |
F1 by page type on the test set (257 articles; 28 to 59 pages of each other type):
@@ -156,12 +179,12 @@ F1 by page type on the test set (257 articles; 28 to 59 pages of each other type
| rs-trafilatura 0.1.1 | 0.942 | 0.740 | 0.732 | 0.714 | 0.829 | 0.953 | 0.801 |
| Trafilatura 2.3.1 | 0.934 | 0.719 | 0.589 | 0.757 | 0.786 | 0.941 | 0.817 |
| Readability.js 0.6 + Turndown | 0.931 | 0.634 | 0.614 | 0.637 | 0.711 | 0.882 | 0.805 |
-| Current Extract rule | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 |
+| Main-region baseline | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 |
-"Current Extract rule" is the Extract API's present `include_markdown`
-behaviour (the page's ``, else its only ``, else ``)
-reproduced with nokodown's converter. See
-[benchmark/compare](benchmark/compare/README.md) to reproduce the table.
+"Main-region baseline" converts the page's ``, else its only
+``, else ``, with nokodown's converter and no main-content
+detection. See [benchmark/compare](benchmark/compare/README.md) to reproduce
+the table.
```sh
bundle exec ruby benchmark/wcxb.rb tmp/corpora/wcxb test
@@ -202,28 +225,15 @@ bundle exec rake benchmark:speed
ruby benchmark/speed.rb DIRECTORY --per-page # any folder of .html or .html.gz files
```
-## Using it in the Extract API
-
-The Extract API currently fills `text_content.markdown` from the page's
-``, its only ``, or the whole body. With this gem:
-
-```ruby
-markdown = Nokodown.to_markdown(html, url: final_url)
-```
-
-`main_content: false` gives the whole-page conversion if a request needs it.
-Truncation to 50,000 characters stays in the engine, and `include_tags` /
-`exclude_tags` can be applied to the HTML before it is passed in.
-
## Development
```sh
bundle install
bundle exec rspec
-NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the version SerpApi runs
+NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the last nokolexbor for Ruby 2.7
```
-Specs use RSpec's `should` syntax, like the main SerpApi app.
+Specs use RSpec's `should` syntax.
## License
@@ -231,5 +241,6 @@ Apache License 2.0. The scoring and cleaning heuristics are adapted from
[Readability.js][mozilla]; see [NOTICE](NOTICE).
[benchmark]: https://github.com/scrapinghub/article-extraction-benchmark
+[extract]: https://serpapi.com/extract-api
[mozilla]: https://github.com/mozilla/readability
[wcxb]: https://github.com/Murrough-Foley/web-content-extraction-benchmark
diff --git a/benchmark/compare/README.md b/benchmark/compare/README.md
index d54415e..d869430 100644
--- a/benchmark/compare/README.md
+++ b/benchmark/compare/README.md
@@ -9,7 +9,7 @@ bundle exec rake benchmark:corpora # WCXB into tmp/corpora/wcxb
W=tmp/corpora/wcxb R=tmp/runs
ruby benchmark/compare/run_nokodown.rb $W $R/nokodown
-ruby benchmark/compare/run_nokodown.rb $W $R/current-extract current
+ruby benchmark/compare/run_nokodown.rb $W $R/main-region baseline
python3 -m venv tmp/venv && tmp/venv/bin/pip install -r benchmark/compare/requirements.txt
tmp/venv/bin/python benchmark/compare/run_python.py trafilatura $W $R/trafilatura
@@ -19,15 +19,15 @@ tmp/venv/bin/python benchmark/compare/run_python.py rs-trafilatura $W $R/rs-traf
for i in 0 1 2 3; do node benchmark/compare/run_readability.js $W $R/readability $i 4 & done; wait
tmp/venv/bin/python benchmark/compare/metrics.py $W $R tmp/metrics.json \
- nokodown rs-trafilatura trafilatura readability current-extract
+ nokodown rs-trafilatura trafilatura readability main-region
```
rs-trafilatura has no macOS wheel; building it needs a Rust toolchain, and on
Python 3.14 also `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1`.
-`current-extract` reproduces the Extract API's current rule (the page's
-``, else its only ``, else ``) using nokodown's converter,
-so it isolates the effect of main-content detection.
+`main-region` is the baseline: the page's ``, else its only ``,
+else ``, converted with nokodown's converter, so it isolates the effect
+of main-content detection.
Besides the benchmark's F1 and snippet rates, `metrics.py` reports what an LLM
pays for: tokens per page (o200k tokenizer), the share of tokens spent on URLs
diff --git a/benchmark/compare/run_nokodown.rb b/benchmark/compare/run_nokodown.rb
index 580d02f..4cc2a51 100644
--- a/benchmark/compare/run_nokodown.rb
+++ b/benchmark/compare/run_nokodown.rb
@@ -2,18 +2,18 @@
# Writes nokodown's Markdown for every WCXB page: OUT//md/.md.
#
-# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [current]
+# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [baseline]
#
-# With "current", it reproduces the Extract API's current rule instead: the
-# page's , else its only , else , converted without
-# main-content detection.
+# With "baseline", it converts the main-region baseline instead: the page's
+# , else its only , else , without main-content
+# detection.
require "fileutils"
require "json"
require "zlib"
require_relative "../../lib/nokodown"
-def current_rule(html, url)
+def main_region(html, url)
doc = Nokolexbor::HTML(Nokodown::Charset.utf8(html))
articles = doc.css("article")
region = doc.at_css("main") || (articles.size == 1 ? articles.first : nil) || doc.at_css("body")
@@ -23,7 +23,7 @@ def current_rule(html, url)
end
wcxb, out, mode = ARGV
-abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [current]" unless wcxb && out
+abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [baseline]" unless wcxb && out
%w[dev test].each do |split|
FileUtils.mkdir_p(File.join(out, split, "md"))
@@ -31,7 +31,7 @@ def current_rule(html, url)
id = File.basename(path, ".json")
url = JSON.parse(File.read(path))["url"]
html = Zlib::GzipReader.open(File.join(wcxb, split, "html", "#{id}.html.gz"), &:read)
- markdown = mode == "current" ? current_rule(html, url) : Nokodown.to_markdown(html, url: url)
+ markdown = mode == "baseline" ? main_region(html, url) : Nokodown.to_markdown(html, url: url)
File.write(File.join(out, split, "md", "#{id}.md"), markdown)
end
end
diff --git a/lib/nokodown/metadata.rb b/lib/nokodown/metadata.rb
index e9244fb..cf26452 100644
--- a/lib/nokodown/metadata.rb
+++ b/lib/nokodown/metadata.rb
@@ -1,6 +1,6 @@
# frozen_string_literal: true
-require "cgi"
+require "cgi/escape"
require "json"
module Nokodown
diff --git a/lib/nokodown/url_resolver.rb b/lib/nokodown/url_resolver.rb
index 46999ce..7547091 100644
--- a/lib/nokodown/url_resolver.rb
+++ b/lib/nokodown/url_resolver.rb
@@ -8,6 +8,8 @@ module Nokodown
# tracking parameters, click redirectors and image resize options cost
# tokens and tell a reader nothing.
class UrlResolver
+ # From Ruby 3.4 URI::DEFAULT_PARSER is the RFC 3986 parser, whose unescape is obsolete.
+ URI_PARSER = defined?(URI::RFC2396_PARSER) ? URI::RFC2396_PARSER : URI::DEFAULT_PARSER
TRACKING_PARAM = /\A(?:utm_[a-z_]+|fbclid|gclid|gclsrc|dclid|gbraid|wbraid|msclkid|yclid|igshid|mc_cid|mc_eid|_hsenc|_hsmi|mkt_tok|__s|
_gl|_ga|srsltid|trk|icid|cmpid|hsCtaTracking|ref_|pd_rd_[a-z]+|pf_rd_[a-z]+)\z/ix.freeze
# Amazon session, search-rank and affiliate parameters; on other sites
@@ -223,7 +225,7 @@ def decode(value)
end
def unescape(value)
- decoded = URI::DEFAULT_PARSER.unescape(value)
+ decoded = URI_PARSER.unescape(value)
decoded if decoded.valid_encoding?
end
diff --git a/nokodown.gemspec b/nokodown.gemspec
index a5a164e..28aec9e 100644
--- a/nokodown.gemspec
+++ b/nokodown.gemspec
@@ -13,8 +13,16 @@ Gem::Specification.new do |spec|
spec.license = "Apache-2.0"
spec.required_ruby_version = ">= 2.7"
- spec.files = Dir["lib/**/*.rb", "LICENSE", "NOTICE", "README.md"]
+ spec.files = Dir["lib/**/*.rb", "CHANGELOG.md", "LICENSE", "NOTICE", "README.md"]
spec.require_paths = ["lib"]
- spec.add_dependency "nokolexbor", ">= 0.6"
+ spec.metadata = {
+ "homepage_uri" => spec.homepage,
+ "source_code_uri" => spec.homepage,
+ "changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md",
+ "bug_tracker_uri" => "#{spec.homepage}/issues",
+ "rubygems_mfa_required" => "true"
+ }
+
+ spec.add_dependency "nokolexbor", "~> 0.6"
end