fix(audit): apply robots.txt groups that name the crawler

[kevin9327]

Problem

The crawler introduces itself as OpenSEO-Audit/1.0 in every request, then
reads robots.txt as an anonymous agent:

const robots = robotsParser(`${origin}/robots.txt`, text);
return {
  isAllowed: (url: string) => robots.isAllowed(url) ?? true,
  ...

robots-parser's isAllowed(url, ua) falls back to the User-agent: * group
when ua is omitted, so any group naming the crawler is dead text. That misses
in both directions:

• A site that blocks us — User-agent: OpenSEO-Audit / Disallow: / — is
  crawled anyway.
• A site that admits us over a blanket block — User-agent: * / Disallow: /
  plus User-agent: OpenSEO-Audit / Allow: / — is not crawled at all. This is
  the one users hit: parseRobotsTxt gates both the discovery seeding
  (siteAuditWorkflowPhases) and the crawl frontier (shouldQueueCrawlLink),
  so the audit stops at the start URL and there is nothing on screen to explain
  why.

Fix

Pass the crawler's own token to isAllowed. robots-parser matches the product
token out of OpenSEO-Audit/1.0, so the same string the fetches send is the
right argument. It was written inline in two places already, so it is now a
named constant in the file and the fetch headers read it too — same string, no
behaviour change there.

Tests

New discovery.test.ts. parseRobotsTxt is exported, pure and deterministic,
so no network or mocks are needed.

Two invariants for the defect and one pin: a robots.txt with only a * group
must behave exactly as before — that is what shows this is a more specific
match, not a widening. It passes before and after.

Against main:

 ✓ falls back to the wildcard group when the crawler is not named
 × honors a group that names the crawler
     AssertionError: expected true to be false
 × honors an allowance the crawler is named in
     AssertionError: expected false to be true

Tests  2 failed | 1 passed (3)

With the fix:

Tests  3 passed (3)

How I tested

Windows 11, Node 22.

npx vitest run src/server/lib/audit/   → 7 files, 72 tests passed
npx tsc --noEmit                       → clean
npx prettier --check <the two files>   → clean
npx oxlint src/server/lib/audit/       → 0 warnings, 0 errors

I did not run the whole suite or e2e.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.