Site Audit: add AEO/agent-readiness checks (llms.txt, AI crawlers in robots.txt, markdown alternates, linked JSON-LD)

[Santofer]

Context

OpenSEO tracks whether AI engines mention a brand (Prompt Explorer / AI visibility). The complementary question the site audit doesn't answer yet: is the site itself readable and citable by AI agents? Two sites with identical rankings can have wildly different agent-readiness, and today none of that surfaces in the audit.

We run audits for several client sites (Morocco + UAE markets) and ended up building these checks outside OpenSEO — proposing them upstream because they feel squarely in scope for "site audits" in 2026. Joost de Valk's recent Astro SEO guide (https://joost.blog/astro-seo-complete-guide/) is a good reference for how mainstream this layer is becoming.

Proposed audit checks (roughly by value/effort)

1. AI crawlers in robots.txt — parse robots.txt against the known AI user-agents (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider…), and report which are blocked, distinguishing purpose: training vs search-grounding vs user-browse. Blocking Google-Extended while wanting AI Overview visibility is a very common self-inflicted wound — we found a chatbot company blocking 8 AI bots via a default Cloudflare managed robots.txt without knowing.
2. llms.txt — presence + basic structural validation (H1 title, blockquote summary, ## sections with markdown links) per https://llmstxt.org/.
3. Markdown alternates — <link rel="alternate" type="text/markdown">, per-page .md URLs, and/or content negotiation on Accept: text/markdown.
4. JSON-LD graph quality — flag flat isolated schema blocks vs a linked @graph with resolvable @id references; check trust-signal properties (publishingPrinciples, copyrightHolder, knowsAbout, SearchAction).
5. schemamap.xml / schema endpoints (NLWeb-adjacent, early-stage — lowest priority).

Each check is a cheap HTTP fetch + parse; no DataForSEO credits involved.

Why in OpenSEO rather than a separate tool

The audit already fetches robots.txt and pages; these checks reuse that plumbing and give the site audit a differentiator none of the incumbent tools (Semrush/Ahrefs site audits) ship today. It also closes the loop with Prompt Explorer: "you're not cited" → "here's why your site isn't citable".

Offer

We have working heuristics for all five checks (currently Python, used in production against real client sites) and are happy to port them to TypeScript inside the audit feature — ideally starting with a focused first PR covering checks 1+2, if a maintainer confirms the direction (status:spec-first?).

[WAHIB-EL-KHADIRI]

I'd like to work on this one. Plan: start with the top two priority checks in a focused first PR — (1) AI crawler directives in robots.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc., grouped by training / search-grounding / user-browse purpose) and (2) llms.txt presence + structural validation per llmstxt.org. Both reuse the existing audit fetching plumbing, no external API credits. Markdown alternates and JSON-LD graph quality could then follow as separate PRs. Sound good?

[Santofer]

Thanks for the go-ahead (and the badseo pointer). I dug through the audit engine to scope this concretely before writing code. Here's how I'd fit it in — flagging the decisions I'd want your call on.

How it maps onto the current engine

• Issue types live in src/shared/audit-issues.ts (AUDIT_ISSUE_TYPES, kebab-case ids + severity + fix copy).
• Per-page checks are pure functions over CrawledPageResult (page-reporters.ts); page-analyzer.ts already parses HTML with cheerio, so new per-page signals just need a couple of extracted fields on CrawledPageResult.
• Site-wide resources are already fetched in discovery.ts (robots.txt + sitemap), and site-level findings run in multipage.ts ("set-queries over crawl data").
• badseo fixtures declare expectedIssues (type-checked against the registry) and the harness asserts each page triggers exactly those — so a fixture and its detector have to land in the same PR, or the harness goes red.

Proposed checks (5, grouped by where the data comes from)

Site-level — fetched once in discovery, evaluated in a new site reporter:

id	severity	trigger
llms-txt-missing	info	no /llms.txt (200)
ai-crawler-blocked	warning	robots.txt Disallow: / for major AI UAs (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, …)
schemamap-missing	info	no /schemamap.xml and no Schemamap: directive in robots.txt

Per-page — extracted in page-analyzer.ts, evaluated in page-reporters.ts:

id	severity	trigger
jsonld-not-linked-graph	info	page has ≥2 isolated JSON-LD blocks with no @graph / no resolvable @id links
markdown-alternate-missing	info	no <link rel="alternate" type="text/markdown"> in <head>

Suggested split (each PR self-contained + harness-green)

1. Site-level AEO checks — discovery fetch of /llms.txt + /schemamap.xml, robots.txt AI-UA parsing, the 3 site issue types, badseo fixtures for each, harness updated.
2. Per-page AEO checks — page-analyzer extraction (JSON-LD blocks, markdown-alternate link), the 2 page issue types, badseo fixtures.

I can also go finer (one check per PR) if you prefer smaller reviews.

Decisions I'd like your call on

1. Severity — I defaulted these to info/warning (they're opportunities, not breakage). Sound right, or do you want ai-crawler-blocked as warning and the rest info (as above)?
2. ai-crawler-blocked scope — flag any blocked major AI UA, or only the search-grounding ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended) since blocking training bots (GPTBot/CCBot) can be a deliberate, legitimate choice? I lean toward the latter to avoid false alarms.
3. jsonld-not-linked-graph — this is the most opinionated one (a flat but valid JSON-LD page isn't strictly "wrong"). Keep it as info, or drop it from v1 and keep the graph as a generation-side nicety rather than an audit finding?
4. AI-crawler UA list — happy to keep it as a small maintained constant in src/shared/; any preference on where it should live?

Once you're good with the shape, I'll open PR 1 (site-level) first. And I'll add the matching badseo fixtures so the demo site exercises each new check.

For more badseo ideas generally (unrelated to AEO): a page serving noindex only via X-Robots-Tag on a non-HTML asset, a hreflang cluster with a missing return tag, a soft-404 (200 status on a "not found" body), and a canonical pointing at a noindex URL — all common real-world traps if they're not already covered.

[WAHIB-EL-KHADIRI]

@Santofer nice scoping — this maps very cleanly onto the audit engine's shape.

Heads up so we don't duplicate work: I already shipped 3 of your 5 proposed checks in #122 (open, awaiting review):

• ai-search-crawlers-blocked / ai-training-crawlers-blocked / ai-user-fetchers-blocked — your ai-crawler-blocked, split by purpose (search-grounding vs. training vs. live user-request bots) rather than one flat warning, since blocking a training bot is a very different signal from blocking a user-request fetcher.
• missing-llms-txt / llms-txt-invalid — your llms-txt-missing, plus structural validation against the llmstxt.org spec (checks it actually starts with an H1, not just that the file exists).
• no-markdown-alternates — your markdown-alternate-missing, implemented as a site-level aggregate (one issue if no crawled page advertises a Markdown alternate) rather than per-page, to avoid spamming a finding almost no site satisfies yet.

All three reuse the existing audit-issues registry / multipage-checks plumbing you identified, same pattern you're describing.

That leaves schemamap-missing and jsonld-not-linked-graph genuinely open — neither is touched by #122. Want to take those two? They're a clean, self-contained slice: schemamap-missing is another site-level fetch-and-check like llms-txt-missing, and jsonld-not-linked-graph is a page-analyzer.ts extraction + page-reporters.ts check like the pattern you already mapped out.

Happy to sync on severity defaults for those two if useful — for the three above I went with info for "absence is common, not an error" cases (missing llms.txt, no markdown alternates) and warning only for something actually broken (llms.txt served but malformed) or a stronger AI-visibility signal (search/user-facing crawlers blocked, vs. info for training-only blocks).

[Santofer]

@WAHIB-EL-KHADIRI took you up on the offer — #204 implements the two checks #122 leaves open: schemamap-missing and jsonld-not-linked-graph. Your split of the crawler check by purpose is better than my flat version, and aggregating markdown alternates site-wide is the right call; no objection to any of #122.

Two things worth knowing since they touch shared ground:

schemamap-missing is the engine's first site-level issue. Everything before it hangs off a page row, so it emits pageId: null with the origin as pageUrl and rides inside the existing multipage-checks step — same checkpoint, and insertIssues is already idempotent, so a retry that re-fetches the root resource can't duplicate rows. finalizeAudit now takes origin + the checkpointed robotsText. If #122 needs the same threading for its llms.txt fetch, we've converged on the same shape and the second PR to land should be a small rebase either way.

File overlap is audit-issues.ts, page-analyzer.ts, types.ts, siteAuditWorkflowPhases.ts — no semantic conflict, just adjacent lines. #122 opened first, so land it first and I'll rebase #204 on top; happy to do that work.

On severity I followed your convention exactly: both are info, since "absence is common, not an error" fits a missing schemamap and a flat-but-valid JSON-LD page equally well.

One extra in #204 worth a look if you like the idea: badseo gets a matched pair of fixtures — a page with three disconnected entities that must fire, and the same three entities joined by @id that must stay clean. The second is the regression guard; without it a linked-graph check quietly starts flagging correct markup. Might be worth the same treatment for llms-txt-invalid (a valid llms.txt that must not fire).