docs: reposition to verifiable web retrieval for AI agents

docs/README.md

@@ -2,14 +2,17 @@
 
 Start here. This page is the entry point for everyone who isn't sure which file to open.
 
+> **Repositioned 2026-05-28** from "open-source Tavily alternative" to "verifiable web retrieval for AI agents." See [`strategy/positioning.md`](./strategy/positioning.md) for the new one-liner and [`strategy/market.md`](./strategy/market.md) for the structural shifts that drove it. ADRs 0009–0013 capture the technical implications.
+
 ## I want to…
 
 ### …understand the product
 
 | For | Read |
 |-----|------|
 | The 30-second pitch | [`/README.md`](../README.md) (repo root) |
-| Why we built UnSearch and for whom | [`strategy/icp.md`](./strategy/icp.md) and [`strategy/positioning.md`](./strategy/positioning.md) |
+| Why we built UnSearch and the problem it solves | [`strategy/market.md`](./strategy/market.md) and [`strategy/positioning.md`](./strategy/positioning.md) |
+| Who we sell to | [`strategy/icp.md`](./strategy/icp.md) |
 | What's shipped vs. in beta vs. planned | [`feature-matrix.md`](./feature-matrix.md) |
 | Pricing rationale | [`strategy/pricing.md`](./strategy/pricing.md) |
 | Where the company is going | [`roadmap.md`](./roadmap.md) and [`strategy/mrr-plan.md`](./strategy/mrr-plan.md) |
@@ -18,10 +21,12 @@ Start here. This page is the entry point for everyone who isn't sure which file
 
 | For | Read |
 |-----|------|
+| 60-second MCP install (lead path) | [`/README.md`](../README.md#quick-start--mcp-first) |
 | 5-minute self-host quickstart | [`quickstart.md`](./quickstart.md) |
-| Migrate from Tavily | [`migration/from-tavily.md`](./migration/from-tavily.md) |
+| Migrate from Tavily (compatibility surface) | [`migration/from-tavily.md`](./migration/from-tavily.md) |
 | Endpoint contracts | [`API_REFERENCE.md`](./API_REFERENCE.md) (or live OpenAPI at `/docs`) |
 | Worked examples per endpoint | [`API_EXAMPLES.md`](./API_EXAMPLES.md) |
+| Citation envelope schema (the wedge primitive) | [`citation-envelope.md`](./citation-envelope.md) |
 | Which AI model runs each request | [`ai-pipeline.md`](./ai-pipeline.md) (and [`ai-quick-reference.md`](./ai-quick-reference.md) for a one-pager) |
 | Use the Python SDK | [`/apps/sdk-py/README.md`](../apps/sdk-py/README.md) |
 | Use the TypeScript SDK | [`/apps/sdk-ts/README.md`](../apps/sdk-ts/README.md) |
@@ -32,7 +37,7 @@ Start here. This page is the entry point for everyone who isn't sure which file
 | For | Read |
 |-----|------|
 | What's where in the repo | [`what-is-what.md`](./what-is-what.md) |
-| How the v2.0 architecture works | [`architecture.md`](./architecture.md) |
+| How the architecture works | [`architecture.md`](./architecture.md) |
 | Cloudflare-specific wiring | [`cloudflare-architecture.md`](./cloudflare-architecture.md) and [`/workers/README.md`](../workers/README.md) |
 | Why we made each major decision | [`adr/`](./adr/README.md) |
 | Repo conventions (testing, commits, naming) | [`/CONTRIBUTING.md`](../CONTRIBUTING.md) and [`/CLAUDE.md`](../CLAUDE.md) |
@@ -42,7 +47,7 @@ Start here. This page is the entry point for everyone who isn't sure which file
 
 | For | Read |
 |-----|------|
-| Deploy to Cloudflare | [`/workers/README.md`](../workers/README.md) and [`deployment/quick-reference.md`](./deployment/quick-reference.md) |
+| Deploy to Cloudflare (recommended) | [`/workers/README.md`](../workers/README.md) and [`deployment/quick-reference.md`](./deployment/quick-reference.md) |
 | Deploy to Railway | [`deployment/railway.md`](./deployment/railway.md) |
 | Deploy to DigitalOcean | [`deployment/digitalocean.md`](./deployment/digitalocean.md) |
 | On-call playbooks | [`operations/RUNBOOKS.md`](./operations/RUNBOOKS.md) |

docs/adr/0009-verifiable-retrieval-as-product-surface.md

@@ -0,0 +1,86 @@
+# ADR-0009: Verifiable Retrieval as the product surface
+
+- Status: Accepted
+- Date: 2026-05-28
+- Deciders: @Rakesh1002
+
+## Context
+
+The original UnSearch positioning ("open-source Tavily alternative, 10× cheaper") was anchored on price + drop-in compatibility for indie developers building agents. Three structural shifts in early 2026 made that wedge unsellable to the indie cohort:
+
+1. **Anthropic shipped native `web_search` as a server-side tool** (default in Sonnet 4.6 / Opus 4.7) with 2026 Dynamic Filtering. Free at usage tier for Claude API customers.
+2. **OpenAI Codex CLI ships first-party web search default-on**, backed by OpenAI's indexed cache.
+3. **The MCP registry crossed 800 official servers and 13K+ total**; Firecrawl, Exa, Tavily, Linkup all ship search MCPs. "MCP search" became table stakes.
+
+Indie developers — the prior wedge — moved to native LLM search at zero cost. The "cheaper Tavily" wedge survives mechanically but no longer differentiates.
+
+At the same time, three regulatory / litigation events opened a different, larger opening:
+
+1. **Q1 2026 US court sanctions for AI-hallucinated legal citations crossed $145,000.** Largest single sanction: $110K, Oregon, April 4 2026.
+2. **Harvey AI ($8B valuation) still hallucinates 1-in-6 queries**, demonstrating that even the leading legal-AI vendor hasn't solved citation grounding at the infra layer.
+3. **EU AI Act Article 12 full enforcement begins August 2026.** Provenance documentation explicitly required; 6-month log retention minimum; 10-year documentation retention; penalty up to €15M or 3% of worldwide turnover.
+
+Regulated buyers (legal-AI startups, medical RAG, fintech research bots, insurance underwriting AI, gov-tech, BigLaw, banks, pharma) **cannot use native LLM search** because (a) citations aren't customer-pinnable, (b) snapshots aren't reproducible, (c) data leaves the customer's perimeter, (d) there is no audit log retention guarantee. They are forced to hand-roll a citation infrastructure stack from Tavily / Exa + Firecrawl / Jina + Playwright + custom NLI grader + Postgres provenance table — 1–2 FTEs of glue per company.
+
+No third-party infrastructure owns "verifiable retrieval for AI agents" as a primitive. Webrecorder owns the archivist workflow primitive (WACZ) but not the agent shape. Tavily/Exa/Brave own the agent shape but not the provenance primitive. Harvey/Hebbia/V7 Go own the application tier but cannot be embedded by ICP-1 builders. Braintrust/Patronus monitor outputs post-hoc but don't fix the input.
+
+## Decision
+
+UnSearch's product surface is **Verifiable Retrieval Infrastructure**, not "search API for agents." The four jobs the product owns:
+
+1. **Source pinning** — content-addressable snapshot of the exact bytes at retrieval time.
+2. **Signed citations** — a signed envelope per result with `{url, sha256, fetched_at, snapshot_key, signature}` that downstream pipelines and auditors can verify.
+3. **Claim verification** — `{claim, source_url} → {supported, evidence_spans, confidence}` with the snapshot re-fetched and graded.
+4. **Replay / audit** — per-API-key audit log of every retrieval, with WACZ-style replay months later.
+
+These four jobs together — not "search" alone — are the product. Search is the substrate.
+
+Practical implementation:
+
+- Every result from `/api/v1/search`, `/api/v1/agent/search`, and the MCP `search` tool returns an inline `citation_envelope`.
+- `/api/v1/verify/citation` returns pinned snapshot + live diff.
+- `/api/v1/verify/claim` returns span-level grading via Workers AI llama-3.3-70b.
+- `/api/v1/audit` exposes the per-API-key audit log.
+- The MCP server exposes `verify_claim` as a first-class tool, not just `search`.
+
+The Tavily-compatible drop-in `/api/v1/agent/search` (ADR-0003) stays as a compatibility surface but is no longer the lead onboarding path. Its ADR status is downgraded from "primary positioning" to "compatibility surface."
+
+## Consequences
+
+We commit to:
+
+- Citation envelope schema as a permanent API contract. Breaking changes require an explicit ADR + v2 envelope co-existing with v1 for ≥12 months. Audit consumers depend on stability.
+- `verify_claim` accuracy as a top-line product KPI tracked publicly via a quarterly "Hallucinated Citation Index" report.
+- An R2 snapshot store on every tier — Free included. Free is not a crippled version of the wedge or the demo collapses.
+- Workers AI llama-3.3-70b as the default grader (ADR-0004 tier "balanced") — changing this affects accuracy + cost simultaneously and requires a co-ADR.
+- WACZ-Auth spec alignment for the envelope format (ADR-0011) so archivist + journalism + reproducibility ecosystems read our snapshots natively.
+
+What we knowingly give up:
+
+- The indie-dev TAM. Native LLM search ate it; trying to compete head-on wastes runway.
+- The "Tavily-compatible" lead message. We keep the endpoint for SEO + customer-continuity, but it sits below the wedge in messaging hierarchy.
+- The "10× cheaper" lead message. The claim is still true as a side-effect of SearXNG + active-CPU billing, but it stops being the hero on landing pages.
+- A larger initial signup pool. Sharper wedge = smaller TAM, higher ACV, higher conversion. This is intentional.
+
+## Alternatives considered
+
+**1. Stay with "cheaper Tavily" positioning.** Rejected: Anthropic / OpenAI native search make this unwinnable for indie devs. Existing customers are not the lever; future cohorts are. The wedge no longer differentiates.
+
+**2. Pivot to a separate verification-only product ("Signoff") layered on top of any search vendor.** Rejected: introducing a separate brand fragments the OSS flywheel and the MCP distribution surface. The verification primitive must live inside the retrieval primitive to demo end-to-end.
+
+**3. Pivot to vertical applications (legal-AI / medical-RAG / fintech-research SaaS).** Rejected: Harvey, Hebbia, Casetext-derivatives, V7 Go already occupy the application tier. UnSearch is infra. Application-tier competition is a different motion, different ACV, different ICP — not adjacent.
+
+**4. Pivot to hallucination monitoring (Braintrust / Patronus / Galileo overlap).** Rejected: monitoring is post-hoc; the wedge here is fixing the input primitive so there is less to monitor. Different category, different sales cycle.
+
+**5. Add verification as a feature on top of the existing "search API" positioning.** Rejected: that framing buries the wedge below a commoditized lead message. The whole product surface must be verifiable retrieval; verification cannot be a checkbox at the bottom of the feature matrix.
+
+## Cross-references
+
+- [`docs/strategy/positioning.md`](../strategy/positioning.md) — new one-liner and pillars
+- [`docs/strategy/market.md`](../strategy/market.md) — Q1 2026 incidents + EU AI Act forcing function
+- [`docs/strategy/icp.md`](../strategy/icp.md) — Priya / David / Anika persona shifts driven by this ADR
+- [ADR-0010](./0010-cloudflare-containers-as-origin-runtime.md) — runtime choice for the verifiable-retrieval implementation
+- [ADR-0011](./0011-wacz-aligned-signed-envelope.md) — envelope format
+- [ADR-0012](./0012-mcp-first-distribution.md) — distribution surface for the new positioning
+- [ADR-0013](./0013-icp-shift-to-regulated-ai.md) — ICP implications
+- [`docs/citation-envelope.md`](../citation-envelope.md) — schema spec

docs/adr/0010-cloudflare-containers-as-origin-runtime.md

@@ -0,0 +1,70 @@
+# ADR-0010: Cloudflare Containers as origin runtime
+
+- Status: Accepted
+- Date: 2026-05-28
+- Deciders: @Rakesh1002
+
+## Context
+
+ADR-0001 committed to Cloudflare-native edge architecture: Workers + D1 + KV + Vectorize + R2 + Queues + Durable Objects. That stack runs the *edge* layer correctly.
+
+The *origin* layer is where ADR-0001 hit a wall. UnSearch depends on SearXNG — a long-running Python process that aggregates 70+ search engines. Workers cannot host SearXNG: confirmed by the SearXNG maintainers ([discussion #4119](https://github.com/searxng/searxng/discussions/4119)) — SearXNG needs persistent compute and longer-lived process state than Workers' execution model provides. The same is largely true for parts of the FastAPI backend that depend on long-lived sessions or large native dependencies.
+
+Until April 2026, the workable answers were:
+
+1. Run SearXNG + FastAPI on a third-party container host (Railway, Fly, DigitalOcean) and have the Worker proxy to it. Adds an external dependency, hurts the "all-on-Cloudflare" story for ICP-2 buyers, and complicates self-host.
+2. Rewrite the FastAPI backend in TypeScript/Hono running on Workers, replacing SearXNG with direct engine API calls. ~3 months of work, throws away a working 93-endpoint backend, and either reintroduces per-vendor lock-in or attempts to reimplement what SearXNG already does.
+3. Keep the Container service binding in `workers/wrangler.toml` commented out (which is the current state — `wrangler.toml:84-90`) and ship nothing.
+
+**Cloudflare Containers reached General Availability on April 13, 2026.** Per the [Cloudflare Containers docs](https://developers.cloudflare.com/containers/) and the [pricing page](https://developers.cloudflare.com/containers/pricing/), the GA brings three properties that change the trade-off:
+
+- **Active-CPU billing** — only the CPU cycles actually consumed are billed; idle containers cost memory + storage only.
+- **Service bindings address containers by hostname** — Workers can reach Containers as named services without explicit IP / DNS discovery.
+- **Thousands of parallel containers per account** with autoscale.
+
+Active-CPU billing is the critical property. SearXNG idle = ~$0. SearXNG handling a search burst = pennies. This eliminates the cost-driven reason for the rewrite-to-Workers path.
+
+## Decision
+
+UnSearch's origin runtime is **Cloudflare Containers**. The FastAPI backend (`app/`) and the SearXNG sidecar run together as a Container deployment, reachable from the Hono Worker edge (`workers/`) via service binding.
+
+Practical implementation:
+
+- `Dockerfile.cloudflare` packages FastAPI + SearXNG together (supervisord-managed) or as adjacent containers wired via internal DNS. The two-container topology is preferred long-term for independent scaling.
+- `wrangler.toml` (root) declares the Container with active-CPU billing.
+- `workers/wrangler.toml:84-90` Container service binding is uncommented; Worker requests resolve to the Container by hostname.
+- Hardcoded `localhost` URLs in `app/config.py:31,37,47,79,140` are replaced with env-driven container-internal DNS.
+- Hosted and self-host topologies use identical Docker images; self-host customers `wrangler containers deploy` from their forked repo.
+
+## Consequences
+
+We commit to:
+
+- Cloudflare as the runtime substrate not just at the edge but at the origin. BYOC support (AWS / GCP / Azure deploy templates) is deferred to Month 10+, and is positioned as a customer accommodation, not as a strategic axis.
+- Treating active-CPU billing as a P0 SLO concern — any code path that spins CPU unnecessarily (chatty polling, busy loops, unbatched engine fan-out) is a cost regression, not just a latency regression.
+- A single set of deployment runbooks (`docs/deployment/`) anchored on `wrangler containers deploy`, with Railway / DigitalOcean kept as alternates for the open-source community but not as the default story.
+- Workers reaching Containers via service binding only — no public Container hostnames, no direct ingress.
+
+What we knowingly give up:
+
+- A multi-cloud-from-day-one self-host story. ICP-2 buyers without Cloudflare strategy must wait until BYOC ships in Month 10+. We mitigate by selling self-host to CF-friendly accounts first.
+- Theoretical edge-perfect latency on hot search paths. SearXNG must be reached, which crosses an internal hop. Mitigated by KV cache + the empirical observation that the 70+ engine fan-out is dominated by third-party engine latency, not by the internal hop.
+
+## Alternatives considered
+
+**1. Rewrite FastAPI to Hono on Workers and replace SearXNG with direct engine calls.** Rejected: ~3 months of work, abandons a production-ready 93-endpoint backend (audit confirms no critical stubs), reintroduces per-engine vendor relationships UnSearch wanted to abstract via SearXNG, and breaks ADR-0002.
+
+**2. Run FastAPI + SearXNG on Railway / Fly / DigitalOcean as the primary topology, with Workers proxying.** Rejected: weakens the all-on-Cloudflare positioning for ICP-2 (self-host on customer's CF account); introduces a third-party billing relationship; complicates the self-host story; loses the active-CPU billing advantage; multi-vendor support contracts add overhead.
+
+**3. Run SearXNG on Containers, FastAPI on Workers (Python via Pyodide / WebAssembly).** Rejected: FastAPI on Pyodide is experimental, has performance footguns, and would force a deeper rewrite of any sync DB code. Not worth the complexity given that Containers can host FastAPI directly.
+
+**4. Wait for Pyodide Workers maturity instead of adopting Containers.** Rejected: that path's timeline is open; Containers GA shipped; we can deploy this month. ADR-0009 commits to a 3-week rebuild that requires a deploy path that exists today.
+
+## Cross-references
+
+- [ADR-0001](./0001-cloudflare-native-edge-architecture.md) — original CF-native commitment that this ADR extends
+- [ADR-0002](./0002-searxng-as-meta-search-aggregator.md) — SearXNG dependency that requires persistent compute
+- [ADR-0009](./0009-verifiable-retrieval-as-product-surface.md) — new product surface this runtime supports
+- [Cloudflare Containers docs](https://developers.cloudflare.com/containers/)
+- [Cloudflare Containers pricing](https://developers.cloudflare.com/containers/pricing/)
+- [SearXNG Workers discussion](https://github.com/searxng/searxng/discussions/4119) — confirms SearXNG cannot run on Workers