The #1 invisible-to-ChatGPT failure mode is a site that blocks AI crawlers in `robots.txt` or via firewall/WAF rules (Cloudflare Bot Fight Mode, AWS WAF AI-bot blocklists). Users don't realize they're blocked until they run the full scanner.
A `doctor` command would diagnose this in 30 seconds without a full scan.
Scope
`aeorank-cli doctor ` should:
- Fetch `robots.txt` and list every AI agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot, …) as allowed / blocked / unspecified.
- Make a HEAD request as each agent's UA and report the status code.
- Flag Cloudflare / AWS WAF signals in response headers (`cf-mitigated`, `x-amzn-waf-action`).
- Check for common firewall blocks on `/llms.txt` specifically (many WAFs serve 403 for unknown paths with .txt extensions).
- Print a clean table and a one-line verdict: "✅ All major AI crawlers can reach your site" or "❌ 4 of 9 AI crawlers blocked — see details".
Acceptance
Meatier than a good-first-issue but still scoped — ideal for someone who wants to ship a distinct feature.
The #1 invisible-to-ChatGPT failure mode is a site that blocks AI crawlers in `robots.txt` or via firewall/WAF rules (Cloudflare Bot Fight Mode, AWS WAF AI-bot blocklists). Users don't realize they're blocked until they run the full scanner.
A `doctor` command would diagnose this in 30 seconds without a full scan.
Scope
`aeorank-cli doctor ` should:
Acceptance
Meatier than a good-first-issue but still scoped — ideal for someone who wants to ship a distinct feature.