From 4561f8344383b342195c3831ab7b2c291b7d4519 Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:00 +0300 Subject: [PATCH 1/7] Install from RubyGems, and name the public API The README pointed at the private repository over SSH. It now installs the published gem, and says that a plain gem install on Ruby 2.7 or 3.0 needs nokolexbor 0.6.4 first: RubyGems picks nokolexbor 0.8, which needs Ruby 3.1, while Bundler resolves to 0.6.4 on its own. Only Nokodown.extract, Nokodown.to_markdown and Nokodown::Result are public; the other classes can change in any release. --- README.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e21f99e..ab4a43f 100644 --- a/README.md +++ b/README.md @@ -14,10 +14,17 @@ Built on [Nokolexbor](https://github.com/serpapi/nokolexbor), for the ## Installation ```ruby -gem "nokodown", git: "git@github.com:serpapi/nokodown.git" +gem "nokodown" ``` -Ruby 2.7 or newer, nokolexbor 0.6 or newer. +Ruby 2.7 or newer, nokolexbor 0.6 or a later 0.x release. nokolexbor 0.7 and +later need Ruby 3.1. Bundler picks 0.6.4 on older Rubies, but `gem install` +does not, so install that version first: + +```sh +gem install nokolexbor -v 0.6.4 +gem install nokodown +``` ## Usage @@ -41,6 +48,9 @@ Nokodown.to_markdown(html, url: "https://example.com/post") # just the Markdown `html` is a string in any encoding. Binary strings are decoded using their ``, and invalid bytes are dropped. +`Nokodown.extract`, `Nokodown.to_markdown` and `Nokodown::Result` are the +public API. Other classes are internal and can change in any release. + | Option | Default | | | --- | --- | --- | | `url` | `nil` | Page URL. Relative links and images are resolved against it (and ``), and links to the page itself are reduced to their text. | From d4511ed31ae0c621e21821c0e8e1a50e37289e25 Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:01 +0300 Subject: [PATCH 2/7] Carry Readability's NOTICE in full Apache-2.0 section 4(d) requires a derivative work to include the attribution notices from its source's NOTICE file. Readability's lists Mozilla and Contributors as well as Arc90, and ours had only Arc90. --- NOTICE | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/NOTICE b/NOTICE index d235277..3506969 100644 --- a/NOTICE +++ b/NOTICE @@ -3,4 +3,8 @@ Copyright 2026 SerpApi The content scoring, conditional cleaning, title and metadata heuristics are adapted from Readability.js (https://github.com/mozilla/readability), -Copyright (c) 2010 Arc90 Inc, licensed under the Apache License, Version 2.0. +licensed under the Apache License, Version 2.0. Its NOTICE reads: + + Readability + Copyright (c) 2010 Arc90 Inc + Copyright (c) 2010-2026 Mozilla and Contributors From c4c4e7b16ec4d92d0a5c1163a7d59a72921c71f9 Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:02 +0300 Subject: [PATCH 3/7] Prepare the gemspec for RubyGems nokolexbor is limited to 0.x, so a 1.0 with breaking changes is not picked up, and gem build no longer warns about an open-ended dependency. The gem links its source, changelog and issues on RubyGems, and pushing a release needs MFA. CHANGELOG.md starts with 0.1.0 and ships in the gem. --- CHANGELOG.md | 12 ++++++++++++ nokodown.gemspec | 12 ++++++++++-- 2 files changed, 22 insertions(+), 2 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..0325dfc --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,12 @@ +# Changelog + +## 0.1.0 (2026-11-02) + +First release. + +- `Nokodown.extract` returns the main content of a web page as Markdown, + with Readability-style metadata (title, byline, excerpt, site name, + published time, language). +- `Nokodown.to_markdown` returns only the Markdown. +- Works on articles and on home, listing, product, documentation, forum and + service pages. diff --git a/nokodown.gemspec b/nokodown.gemspec index a5a164e..28aec9e 100644 --- a/nokodown.gemspec +++ b/nokodown.gemspec @@ -13,8 +13,16 @@ Gem::Specification.new do |spec| spec.license = "Apache-2.0" spec.required_ruby_version = ">= 2.7" - spec.files = Dir["lib/**/*.rb", "LICENSE", "NOTICE", "README.md"] + spec.files = Dir["lib/**/*.rb", "CHANGELOG.md", "LICENSE", "NOTICE", "README.md"] spec.require_paths = ["lib"] - spec.add_dependency "nokolexbor", ">= 0.6" + spec.metadata = { + "homepage_uri" => spec.homepage, + "source_code_uri" => spec.homepage, + "changelog_uri" => "#{spec.homepage}/blob/main/CHANGELOG.md", + "bug_tracker_uri" => "#{spec.homepage}/issues", + "rubygems_mfa_required" => "true" + } + + spec.add_dependency "nokolexbor", "~> 0.6" end From 0722fb2a5c99f699bc53f74999b6bfaa283566bb Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:03 +0300 Subject: [PATCH 4/7] Remove internal Extract API and production notes The README described how to wire the gem into the Extract API and benchmarked the API's current rule by name. The integration notes belong with that change, not in a public README, and the baseline is now named for what it does: convert the page's
, else its only
, else , without main-content detection. Its scores are unchanged; run_nokodown.rb takes "baseline" instead of "current". Comments that described SerpApi's production Ruby and nokolexbor versions now say why CI tests them. --- .github/workflows/ci.yml | 2 +- Gemfile | 2 +- README.md | 29 ++++++++--------------------- benchmark/compare/README.md | 10 +++++----- benchmark/compare/run_nokodown.rb | 14 +++++++------- 5 files changed, 22 insertions(+), 35 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3b72486..095b67a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: fail-fast: false matrix: include: - # SerpApi runs Ruby 2.7.8 with nokolexbor 0.6.4. + # The oldest supported Ruby, with the last nokolexbor that runs on it. - ruby: "2.7" nokolexbor: "0.6.4" - ruby: "3.4" diff --git a/Gemfile b/Gemfile index f258dbb..dd733a3 100644 --- a/Gemfile +++ b/Gemfile @@ -4,7 +4,7 @@ source "https://rubygems.org" gemspec -# CI pins the version SerpApi runs in production (NOKOLEXBOR_VERSION=0.6.4). +# CI also tests nokolexbor 0.6.4, the last release for Ruby 2.7 (NOKOLEXBOR_VERSION=0.6.4). nokolexbor_version = ENV["NOKOLEXBOR_VERSION"].to_s gem "nokolexbor", nokolexbor_version unless nokolexbor_version.empty? diff --git a/README.md b/README.md index ab4a43f..1306c92 100644 --- a/README.md +++ b/README.md @@ -156,7 +156,7 @@ point. | rs-trafilatura 0.1.1 | 0.838 | 0.873 | 12.7% | 87.7% | 2,664 | 18% | | Trafilatura 2.3.1 | 0.816 | 0.858 | 8.1% | 81.8% | 2,552 | 17% | | Readability.js 0.6 + Turndown | 0.770 | 0.830 | 10.0% | 76.5% | 2,899 | 23% | -| Current Extract rule | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% | +| Main-region baseline | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% | F1 by page type on the test set (257 articles; 28 to 59 pages of each other type): @@ -166,12 +166,12 @@ F1 by page type on the test set (257 articles; 28 to 59 pages of each other type | rs-trafilatura 0.1.1 | 0.942 | 0.740 | 0.732 | 0.714 | 0.829 | 0.953 | 0.801 | | Trafilatura 2.3.1 | 0.934 | 0.719 | 0.589 | 0.757 | 0.786 | 0.941 | 0.817 | | Readability.js 0.6 + Turndown | 0.931 | 0.634 | 0.614 | 0.637 | 0.711 | 0.882 | 0.805 | -| Current Extract rule | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 | +| Main-region baseline | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 | -"Current Extract rule" is the Extract API's present `include_markdown` -behaviour (the page's `
`, else its only `
`, else ``) -reproduced with nokodown's converter. See -[benchmark/compare](benchmark/compare/README.md) to reproduce the table. +"Main-region baseline" converts the page's `
`, else its only +`
`, else ``, with nokodown's converter and no main-content +detection. See [benchmark/compare](benchmark/compare/README.md) to reproduce +the table. ```sh bundle exec ruby benchmark/wcxb.rb tmp/corpora/wcxb test @@ -212,28 +212,15 @@ bundle exec rake benchmark:speed ruby benchmark/speed.rb DIRECTORY --per-page # any folder of .html or .html.gz files ``` -## Using it in the Extract API - -The Extract API currently fills `text_content.markdown` from the page's -`
`, its only `
`, or the whole body. With this gem: - -```ruby -markdown = Nokodown.to_markdown(html, url: final_url) -``` - -`main_content: false` gives the whole-page conversion if a request needs it. -Truncation to 50,000 characters stays in the engine, and `include_tags` / -`exclude_tags` can be applied to the HTML before it is passed in. - ## Development ```sh bundle install bundle exec rspec -NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the version SerpApi runs +NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec # the last nokolexbor for Ruby 2.7 ``` -Specs use RSpec's `should` syntax, like the main SerpApi app. +Specs use RSpec's `should` syntax. ## License diff --git a/benchmark/compare/README.md b/benchmark/compare/README.md index d54415e..d869430 100644 --- a/benchmark/compare/README.md +++ b/benchmark/compare/README.md @@ -9,7 +9,7 @@ bundle exec rake benchmark:corpora # WCXB into tmp/corpora/wcxb W=tmp/corpora/wcxb R=tmp/runs ruby benchmark/compare/run_nokodown.rb $W $R/nokodown -ruby benchmark/compare/run_nokodown.rb $W $R/current-extract current +ruby benchmark/compare/run_nokodown.rb $W $R/main-region baseline python3 -m venv tmp/venv && tmp/venv/bin/pip install -r benchmark/compare/requirements.txt tmp/venv/bin/python benchmark/compare/run_python.py trafilatura $W $R/trafilatura @@ -19,15 +19,15 @@ tmp/venv/bin/python benchmark/compare/run_python.py rs-trafilatura $W $R/rs-traf for i in 0 1 2 3; do node benchmark/compare/run_readability.js $W $R/readability $i 4 & done; wait tmp/venv/bin/python benchmark/compare/metrics.py $W $R tmp/metrics.json \ - nokodown rs-trafilatura trafilatura readability current-extract + nokodown rs-trafilatura trafilatura readability main-region ``` rs-trafilatura has no macOS wheel; building it needs a Rust toolchain, and on Python 3.14 also `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1`. -`current-extract` reproduces the Extract API's current rule (the page's -`
`, else its only `
`, else ``) using nokodown's converter, -so it isolates the effect of main-content detection. +`main-region` is the baseline: the page's `
`, else its only `
`, +else ``, converted with nokodown's converter, so it isolates the effect +of main-content detection. Besides the benchmark's F1 and snippet rates, `metrics.py` reports what an LLM pays for: tokens per page (o200k tokenizer), the share of tokens spent on URLs diff --git a/benchmark/compare/run_nokodown.rb b/benchmark/compare/run_nokodown.rb index 580d02f..4cc2a51 100644 --- a/benchmark/compare/run_nokodown.rb +++ b/benchmark/compare/run_nokodown.rb @@ -2,18 +2,18 @@ # Writes nokodown's Markdown for every WCXB page: OUT//md/.md. # -# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [current] +# ruby benchmark/compare/run_nokodown.rb WCXB_DIR OUT_DIR [baseline] # -# With "current", it reproduces the Extract API's current rule instead: the -# page's
, else its only
, else , converted without -# main-content detection. +# With "baseline", it converts the main-region baseline instead: the page's +#
, else its only
, else , without main-content +# detection. require "fileutils" require "json" require "zlib" require_relative "../../lib/nokodown" -def current_rule(html, url) +def main_region(html, url) doc = Nokolexbor::HTML(Nokodown::Charset.utf8(html)) articles = doc.css("article") region = doc.at_css("main") || (articles.size == 1 ? articles.first : nil) || doc.at_css("body") @@ -23,7 +23,7 @@ def current_rule(html, url) end wcxb, out, mode = ARGV -abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [current]" unless wcxb && out +abort "usage: ruby run_nokodown.rb WCXB_DIR OUT_DIR [baseline]" unless wcxb && out %w[dev test].each do |split| FileUtils.mkdir_p(File.join(out, split, "md")) @@ -31,7 +31,7 @@ def current_rule(html, url) id = File.basename(path, ".json") url = JSON.parse(File.read(path))["url"] html = Zlib::GzipReader.open(File.join(wcxb, split, "html", "#{id}.html.gz"), &:read) - markdown = mode == "current" ? current_rule(html, url) : Nokodown.to_markdown(html, url: url) + markdown = mode == "baseline" ? main_region(html, url) : Nokodown.to_markdown(html, url: url) File.write(File.join(out, split, "md", "#{id}.md"), markdown) end end From 9a6a8b7c20630b4ee63654d4231532702d631bdf Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:04 +0300 Subject: [PATCH 5/7] Stop calling the WCXB test set held out Some test pages were studied while tuning nokodown, and 139 of the 511 have the same HTML as a development page, so the test scores are not a blind result. The README now says so. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1306c92..517c731 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,9 @@ region, or most of the page's text, is never removed because of its class name. [WCXB][wcxb] has 2,008 human-reviewed pages: 1,050 articles and 958 product, collection, listing, forum, documentation and service (SaaS, pricing) pages, -split into a development set (1,497) and a held-out test set (511). Scores use +split into a development set (1,497) and a test set (511). The test set was +not held out: some of its pages were studied while tuning nokodown, and 139 +of them have the same HTML as a development page. Scores use its metric: word-level F1 against the reference main content, plus the share of "must include" snippets found and of boilerplate snippets leaked. All systems ran on the same pages with links, images and tables on; their Markdown From 38f9037c4e758fb83f9cfaf982e934ce385dcee9 Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:05 +0300 Subject: [PATCH 6/7] Test on Ruby 4.0 and silence its two warnings Ruby 4.0 warns that the CGI library is gone; cgi/escape has CGI.unescapeHTML on every Ruby from 2.7. From 3.4 URI::DEFAULT_PARSER is the RFC 3986 parser, whose unescape is obsolete, so the resolver uses the RFC 2396 parser where it exists. Both warnings showed only with -W. The specs pass without warnings on Ruby 2.7, 3.4 and 4.0. --- .github/workflows/ci.yml | 2 ++ lib/nokodown/metadata.rb | 2 +- lib/nokodown/url_resolver.rb | 4 +++- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 095b67a..e1462ce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,6 +20,8 @@ jobs: nokolexbor: "0.6.4" - ruby: "3.4" nokolexbor: "" + - ruby: "4.0" + nokolexbor: "" env: NOKOLEXBOR_VERSION: ${{ matrix.nokolexbor }} steps: diff --git a/lib/nokodown/metadata.rb b/lib/nokodown/metadata.rb index e9244fb..cf26452 100644 --- a/lib/nokodown/metadata.rb +++ b/lib/nokodown/metadata.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require "cgi" +require "cgi/escape" require "json" module Nokodown diff --git a/lib/nokodown/url_resolver.rb b/lib/nokodown/url_resolver.rb index 46999ce..7547091 100644 --- a/lib/nokodown/url_resolver.rb +++ b/lib/nokodown/url_resolver.rb @@ -8,6 +8,8 @@ module Nokodown # tracking parameters, click redirectors and image resize options cost # tokens and tell a reader nothing. class UrlResolver + # From Ruby 3.4 URI::DEFAULT_PARSER is the RFC 3986 parser, whose unescape is obsolete. + URI_PARSER = defined?(URI::RFC2396_PARSER) ? URI::RFC2396_PARSER : URI::DEFAULT_PARSER TRACKING_PARAM = /\A(?:utm_[a-z_]+|fbclid|gclid|gclsrc|dclid|gbraid|wbraid|msclkid|yclid|igshid|mc_cid|mc_eid|_hsenc|_hsmi|mkt_tok|__s| _gl|_ga|srsltid|trk|icid|cmpid|hsCtaTracking|ref_|pd_rd_[a-z]+|pf_rd_[a-z]+)\z/ix.freeze # Amazon session, search-rank and affiliate parameters; on other sites @@ -223,7 +225,7 @@ def decode(value) end def unescape(value) - decoded = URI::DEFAULT_PARSER.unescape(value) + decoded = URI_PARSER.unescape(value) decoded if decoded.valid_encoding? end From 602b036448e1a3635bfed155b1ebf177e2b4a155 Mon Sep 17 00:00:00 2001 From: Jasir Zaeem Date: Fri, 30 Oct 2026 16:58:06 +0300 Subject: [PATCH 7/7] Add the SerpApi logo and point to the Extract API The README opens with the SerpApi logo from the media kit, using the gradient version, which reads on GitHub's light and dark themes. A note under the intro links the Extract API for readers who would rather not fetch pages themselves. --- README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 517c731..9290a45 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,9 @@ +

+ + SerpApi + +

+ # nokodown Any web page to main-content Markdown, at Nokolexbor speed. @@ -8,8 +14,13 @@ language). It works on articles and on everything else: home pages, search results, product and pricing pages, documentation, forums. Built on [Nokolexbor](https://github.com/serpapi/nokolexbor), for the -`include_markdown` option of SerpApi's Extract API. The name is Nokolexbor's -"noko" plus Markdown's "down". +`include_markdown` option of SerpApi's [Extract API][extract]. The name is +Nokolexbor's "noko" plus Markdown's "down". + +> **Don't want to fetch the pages yourself?** SerpApi's [Extract API][extract] +> takes a URL and returns the page's Markdown as JSON, along with its meta +> tags, structured data, links and images. +> [Try the Extract API][extract]. ## Installation @@ -230,5 +241,6 @@ Apache License 2.0. The scoring and cleaning heuristics are adapted from [Readability.js][mozilla]; see [NOTICE](NOTICE). [benchmark]: https://github.com/scrapinghub/article-extraction-benchmark +[extract]: https://serpapi.com/extract-api [mozilla]: https://github.com/mozilla/readability [wcxb]: https://github.com/Murrough-Foley/web-content-extraction-benchmark