feat(map): return discovered sitemap document URLs - #459
Merged
Conversation
/v1/map walked the sitemap tree during discovery but kept only the
<url><loc> page entries, so the sitemap documents themselves were never
visible to callers. They are already in hand at that point, so surfacing
them costs no extra fetch.
- fetch_sitemap_tree returns SitemapWalk { pages, sitemaps }; only
documents that answered with parseable content are listed, so a 404
fallback guess is never reported as a sitemap of the site
- MapData gains a sitemaps array, kept out of links because a sitemap
file is not a page; empty when useSitemap is false
- CLI prints sitemaps to stderr in text mode so stdout stays a pipeable
page list, and adds the field to --format json
- the Firecrawl-compat /v2 map shape is unchanged
- docs/features/map documented /v1/map with the /v2 envelope and
parameters that /v1 does not accept; its example would have raised
KeyError against the real response
ref #440
Review found three ways the new sitemaps list told the caller something untrue. - SitemapResult::is_empty() collapsed "could not fetch" and "fetched a valid sitemap that currently lists nothing", so a real /sitemap.xml returning 200 with an empty <urlset> was silently absent. The signal is now whether the document was retrieved and recognised as a sitemap. Fixed in fetch_sitemap_raw so every caller gets it: a 2xx body that yields no entries and declares no sitemap root is not a sitemap and no longer parses as one. Page discovery is unaffected; both arms already produced an empty result. - The embedded MCP crw_map handler builds its JSON by hand and had diverged from the REST route, so an agent never saw the field the same request returns over HTTP. - Docs claimed more than the walk delivers: the list is partial when a cap or the deadline stops discovery early, and within a level the order is completion order, not a stable BFS order. llms-full.txt also attributed the MCP layer's truncated/totalDiscovered markers to the REST endpoint, which never emits them. ref #440
crw_map's output bounding existed to keep a large map out of the model's context, but it only ever looked at links. A site with a deep sitemap index can list thousands of sitemap documents, so passing that array through untouched reopened the hole the cap was there to close. Both arrays now share the map limit and are bounded independently: a short link list no longer exempts a long sitemap list. A cut sitemap list reports totalSitemaps next to the existing truncated marker, so the model is never silently handed a partial list. ref #440
The crw-map skill is the fifth surface that documents map output and was the one left behind: it still described the REST response as links-only and the MCP bound as a 100-URL cap, so an agent reading it would not know the sitemaps list exists or that it can be truncated too. Also corrects bound_map_links' doc comment, which still claimed to truncate only links. ref #440
us
force-pushed
the
feat/map-sitemap-files
branch
from
August 24, 2026 10:42
5152378 to
2a66288
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/v1/mapwalked the sitemap tree during discovery but returned only the<url><loc>page entries. The sitemap documents themselves were dropped on theway out, so a caller had no way to get
/sitemap.xml,/sitemap_index.xmlor/product-sitemap.xmlback. They are already in hand at that point, sosurfacing them costs no extra fetch.
What changed
fetch_sitemap_treereturnsSitemapWalk { pages, sitemaps }. Only documentsthat answered with parseable content are listed, so a fallback path that 404s
(
/wp-sitemap.xmlon a non-WordPress site) is never reported as a sitemap ofthe site.
MapDatagainssitemaps: string[]. Kept out oflinksbecause a sitemapfile is not a page, and merging them would silently change the output of every
existing caller. Empty when
useSitemapisfalse.Sitemap: <url>lines to stderr so stdout stays apipeable page list;
--format jsongains the field.map.md,response-shapes.md,llms-full.txt, both OpenAPI specs(the embedded
crates/crw-server/openapi/copies and thedocs/copies), andthe regenerated static pages.
docs/features/map/index.htmldocumented/v1/mapwith the/v2envelope(top-level
links) and parameters/v1does not accept (search,includeSubdomains,ignoreSitemap). Its Python example would have raisedKeyErroragainst the real response. Corrected in the same pass.Compatibility
The Firecrawl-compat
/v2/mapshape is untouched: it builds its ownV2MapResponse/V2Linktypes and does not readMapData. Verified against arunning server: top-level keys stay
["links", "success"]and each link objectstays
{"url": ...}.sitemapsis additive on/v1, so existing/v1callers are unaffected.Verification
Against a local server on the built binary:
POST /v1/map {"url": "https://wordpress.org", "crawlFallback": false, "limit": 5}returns 27 sitemaps (
sitemap.xml,news-sitemap.xml,themes/sitemap.xml,plugins/sitemap-index-1.xml, ...) alongside the page links.POST /v1/map {"useSitemap": false}returns"sitemaps": [].POST /v2/mapshape unchanged.python.org:/sitemap.xml404, noSitemap:line inrobots.txt) returns an empty list rather than a fabricated one.
New unit test covers the index-plus-child case and asserts a 404 seed is not
reported. OpenAPI drift guard run against a release build:
live /openapi.json matches docs/openapi.json.ref #440