Skip to content

feat(map): return discovered sitemap document URLs - #459

Merged
us merged 4 commits into
mainfrom
feat/map-sitemap-files
Aug 24, 2026
Merged

feat(map): return discovered sitemap document URLs#459
us merged 4 commits into
mainfrom
feat/map-sitemap-files

Conversation

@us

@us us commented Aug 23, 2026

Copy link
Copy Markdown
Owner

/v1/map walked the sitemap tree during discovery but returned only the
<url><loc> page entries. The sitemap documents themselves were dropped on the
way out, so a caller had no way to get /sitemap.xml, /sitemap_index.xml or
/product-sitemap.xml back. They are already in hand at that point, so
surfacing them costs no extra fetch.

What changed

  • fetch_sitemap_tree returns SitemapWalk { pages, sitemaps }. Only documents
    that answered with parseable content are listed, so a fallback path that 404s
    (/wp-sitemap.xml on a non-WordPress site) is never reported as a sitemap of
    the site.
  • MapData gains sitemaps: string[]. Kept out of links because a sitemap
    file is not a page, and merging them would silently change the output of every
    existing caller. Empty when useSitemap is false.
  • CLI: text mode prints Sitemap: <url> lines to stderr so stdout stays a
    pipeable page list; --format json gains the field.
  • Docs: map.md, response-shapes.md, llms-full.txt, both OpenAPI specs
    (the embedded crates/crw-server/openapi/ copies and the docs/ copies), and
    the regenerated static pages.
  • docs/features/map/index.html documented /v1/map with the /v2 envelope
    (top-level links) and parameters /v1 does not accept (search,
    includeSubdomains, ignoreSitemap). Its Python example would have raised
    KeyError against the real response. Corrected in the same pass.

Compatibility

The Firecrawl-compat /v2/map shape is untouched: it builds its own
V2MapResponse / V2Link types and does not read MapData. Verified against a
running server: top-level keys stay ["links", "success"] and each link object
stays {"url": ...}.

sitemaps is additive on /v1, so existing /v1 callers are unaffected.

Verification

Against a local server on the built binary:

  • POST /v1/map {"url": "https://wordpress.org", "crawlFallback": false, "limit": 5}
    returns 27 sitemaps (sitemap.xml, news-sitemap.xml, themes/sitemap.xml,
    plugins/sitemap-index-1.xml, ...) alongside the page links.
  • POST /v1/map {"useSitemap": false} returns "sitemaps": [].
  • POST /v2/map shape unchanged.
  • A site with no sitemap (python.org: /sitemap.xml 404, no Sitemap: line in
    robots.txt) returns an empty list rather than a fabricated one.

New unit test covers the index-plus-child case and asserts a 404 seed is not
reported. OpenAPI drift guard run against a release build:
live /openapi.json matches docs/openapi.json.

ref #440

us added 4 commits August 24, 2026 13:42
/v1/map walked the sitemap tree during discovery but kept only the
<url><loc> page entries, so the sitemap documents themselves were never
visible to callers. They are already in hand at that point, so surfacing
them costs no extra fetch.

- fetch_sitemap_tree returns SitemapWalk { pages, sitemaps }; only
  documents that answered with parseable content are listed, so a 404
  fallback guess is never reported as a sitemap of the site
- MapData gains a sitemaps array, kept out of links because a sitemap
  file is not a page; empty when useSitemap is false
- CLI prints sitemaps to stderr in text mode so stdout stays a pipeable
  page list, and adds the field to --format json
- the Firecrawl-compat /v2 map shape is unchanged
- docs/features/map documented /v1/map with the /v2 envelope and
  parameters that /v1 does not accept; its example would have raised
  KeyError against the real response

ref #440
Review found three ways the new sitemaps list told the caller something
untrue.

- SitemapResult::is_empty() collapsed "could not fetch" and "fetched a
  valid sitemap that currently lists nothing", so a real /sitemap.xml
  returning 200 with an empty <urlset> was silently absent. The signal is
  now whether the document was retrieved and recognised as a sitemap.
  Fixed in fetch_sitemap_raw so every caller gets it: a 2xx body that
  yields no entries and declares no sitemap root is not a sitemap and no
  longer parses as one. Page discovery is unaffected; both arms already
  produced an empty result.
- The embedded MCP crw_map handler builds its JSON by hand and had
  diverged from the REST route, so an agent never saw the field the same
  request returns over HTTP.
- Docs claimed more than the walk delivers: the list is partial when a
  cap or the deadline stops discovery early, and within a level the order
  is completion order, not a stable BFS order. llms-full.txt also
  attributed the MCP layer's truncated/totalDiscovered markers to the
  REST endpoint, which never emits them.

ref #440
crw_map's output bounding existed to keep a large map out of the model's
context, but it only ever looked at links. A site with a deep sitemap
index can list thousands of sitemap documents, so passing that array
through untouched reopened the hole the cap was there to close.

Both arrays now share the map limit and are bounded independently: a
short link list no longer exempts a long sitemap list. A cut sitemap list
reports totalSitemaps next to the existing truncated marker, so the
model is never silently handed a partial list.

ref #440
The crw-map skill is the fifth surface that documents map output and was
the one left behind: it still described the REST response as links-only
and the MCP bound as a 100-URL cap, so an agent reading it would not know
the sitemaps list exists or that it can be truncated too.

Also corrects bound_map_links' doc comment, which still claimed to
truncate only links.

ref #440
@us
us force-pushed the feat/map-sitemap-files branch from 5152378 to 2a66288 Compare August 24, 2026 10:42
@us
us merged commit 07065a4 into main Aug 24, 2026
12 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 24, 2026
@us
us deleted the feat/map-sitemap-files branch August 24, 2026 10:55
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant