feat(audit): AI readiness checks — robots.txt AI crawlers, llms.txt, and Markdown alternates (part 1 of #99)

[WAHIB-EL-KHADIRI]

Part of #99 — now covers three of the four priority checks from the issue: AI crawlers in robots.txt, llms.txt validation, and Markdown alternates. JSON-LD graph quality can follow as a separate PR (it's the lowest-priority item and touches page-analyzer structured-data parsing, a bigger diff on its own).

What this adds

Six new issue types in the shared registry, all site-level (pageId: null — the schema already allows it):

Issue	Severity	When
ai-search-crawlers-blocked	warning	robots.txt blocks index-building agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
ai-user-fetchers-blocked	warning	robots.txt blocks user-request fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User)
ai-training-crawlers-blocked	info	robots.txt blocks training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider)
missing-llms-txt	info	no /llms.txt (absence is common — heads-up, not an error)
llms-txt-invalid	warning	/llms.txt served but violates the spec's one hard requirement (H1 title first)
no-markdown-alternates	info	no crawled page advertises <link rel="alternate" type="text/markdown"> (absence is common — heads-up, not an error)

Design decisions

• Only AI-specific robots.txt blocks are flagged. An agent counts as blocked only when it is treated worse than the generic * rules (checked via a probe UA that falls through to *). A site with a blanket Disallow: / made a site-wide choice — flagging every AI agent there would be noise.
• One issue per purpose group, not per agent — blocked agents are listed in details.blockedAgents with vendor names, so a site blocking 7 training crawlers gets 1 row, not 7.
• Google-Extended nuance from the issue ("blocking Google-Extended while wanting AI Overview visibility") is addressed in the descriptor text: it is listed under training but the explanation calls out that it also gates Gemini answer grounding.
• llms.txt false-positive protection: SPA catch-alls that answer 200 with the app shell for any path are detected (content-type / <!doctype sniff) and treated as missing, not invalid; unreachable fetches stay silent instead of mis-reporting. Structure validation is deliberately lenient — the spec's only hard requirement is the H1 title; optional blockquote/sections are not flagged.
• Markdown alternates is a site-level aggregate, not per-page: almost no site does this today, so flagging every crawled page would be noise. page-analyzer.ts detects the <link rel=alternate type="text/markdown"> tag during the existing per-page cheerio pass (same pattern as the neighboring hreflang extraction) — no extra fetches. The flag flows transiently through CrawledPageResult → the crawl step's slim StepPageSummary (same path already used for title/statusCode), so it needed no new persisted column or schema migration — the finding is only ever recorded as an audit_issues row.
• Plumbing reuse throughout: everything runs in the existing multipage-checks finalize step and reuses the robots.txt text the discovery phase already checkpoints (replay-safe). The only new network cost is a single llms.txt fetch. No external API credits — the shared registry automatically covers the issues UI, CSV export, and MCP.

Testing

• 19 unit tests in ai-readiness.test.ts: purpose grouping, blanket-block silence, subpath-only blocks ignored, dedupe keys, llms.txt valid/empty/no-H1/missing/unreachable cases, markdown-alternates empty-crawl/found/none-found cases.
• tsc --noEmit, oxlint, knip, prettier all clean. Full vitest suite (src/server/lib/audit): 58/58 passing.

🤖 Generated with Claude Code

[sorcerai]

Reviewed exact head against upstream f569726: there is no duplicate ai-crawler-analysis implementation on upstream main. Three blockers remain. (1) /llms.txt uses redirect-following fetch with no final-origin validation and reads response.text() unbounded; prevent cross-origin redirects and use the existing bounded reader. (2) no-markdown is emitted when every crawled page is redirect/non-HTML/blocked/error because those states collapse to hasMarkdownAlternate=false; evaluate only analyzable successful HTML pages or preserve unknown. (3) blockedAgents is an array of objects, but IssuesView joins arrays and renders [object Object]; make details UI-compatible or update the renderer. Add integration coverage for redirects/body bounds and non-analyzable markdown cases, then rerun CI on current main.

[WAHIB-EL-KHADIRI]

Thanks for the thorough review @sorcerai — all three addressed in abb9fc6:

1. llms.txt SSRF + unbounded read: switched to redirect: "manual", reject cross-origin redirects, and re-validate same-origin hops via normalizeAndValidateStartUrl. Body now goes through the shared readBoundedText reader instead of response.text().
2. no-markdown false positive: redirect/blocked/error/non-HTML pages default hasMarkdownAlternate=false, so the check now counts only analyzed HTML pages (new isHtml flag on StepPageSummary) and stays silent when none were analyzable.
3. [object Object]: blockedAgents is now a string[] ("GPTBot (OpenAI)"), compatible with IssuesView's array renderer.

Added integration coverage for cross-origin/same-origin redirects, the HTML app-shell case, the body-size bound, and the non-analyzable markdown cases. Typecheck + lint clean; server audit/workflow tests green.

[WAHIB-EL-KHADIRI]

Thanks for the thorough review — reworked and rebased onto current main.

Rebase / scope: main has since removed StepPageSummary, which the Markdown-alternates check depended on. Rather than re-plumb that onto the new crawl shape in this PR, I split it out — this PR is now the two self-contained checks (robots.txt AI-crawler treatment + llms.txt). That also resolves blocker (2): the Markdown check that emitted false positives on non-analyzable pages is no longer part of this PR.

Blocker (1) — SSRF + unbounded read on /llms.txt: fetchLlmsTxt now uses redirect: "manual", follows at most one hop, rejects any cross-origin redirect, and re-validates the destination through normalizeAndValidateStartUrl (private-IP/DNS guard). The body is read through the shared readBoundedText (exported from scrape.ts) so an oversized third-party file can't be buffered unbounded.

Blocker (3) — [object Object] in IssuesView: blockedAgents is now a flat string ("GPTBot (OpenAI), ClaudeBot (Anthropic)"), so the detail renders as-is with no renderer change.

Tests: added integration coverage for the redirect paths (same-origin follow, cross-origin refusal), the body-bound cap, and the HTML catch-all. Locally: the module's tests pass, tsc is clean for the touched files, and oxlint is clean.

[bensenescu]

Hey,

Thanks for contributing!

For the next few months, I'm going to stop accepting external PRs. Reviewing and testing them has been slowing down progress against the roadmap.

Right now, the best way to contribute is through creating quality "Issues" which are easier for me to review and prioritize right now.

Here is our updated contributing guide: https://github.com/every-app/open-seo/blob/main/docs/CONTRIBUTING.md

Converting, this PR to an issue would be the best way to get it prioritized.

You can use the the /simple-issue-description skill described in the guide to convert the PR to an issue.

[WAHIB-EL-KHADIRI]

Understood - closing rather than leaving it open against the new process.

Not filing a separate issue for this one: #99 already describes the feature and is still open, and this PR was only part 1 of it. The branch stays up if the implementation is useful as a reference for that issue.