Policai is an Australian AI policy tracker. It maintains a curated register of AI policy, regulation, governance and court guidance across federal and state/territory jurisdictions, and automatically detects new developments from official government sources every day.
Product surface:
- searchable policy register with status lifecycle (active, superseded, closed, …)
- developments feed of newly detected policy activity, with provenance and confidence labels
- court AI guidance view
- agencies directory
- interactive Australia map
- timeline, network, and DTA framework visualisations
- MDX-backed blog
Git is the database. The canonical data lives in this repository:
data/policies.json— the curated policy register (only changed by reviewed commits; served through a filtered route)data/developments.json— the automated radar feed, combined at read time with verified legacy announcements from the editorial chronologypublic/data/meta.json— public collection health metadatadata/dta-ai-policy-framework.json— editorial visualization artifact gated by its related policydata/timeline.json,agencies.json,commonwealth-agencies.json— editorial datasets whose public JSON routes apply verification filtersdata/watch-state.json— retryable candidate and source-snapshot statedata/source-reviews.json— detections staged for curated reviewdata/source-monitoring.json— the manual-source review ledger
The maintainer's server runs the collector daily, from its own checkout, over the official sources that reliably permit machine retrieval. Sources protected by browser challenges are kept in the same source catalogue but reviewed through the manual coverage ledger. Candidate pages from browser-only sources are retrieved through a self-hosted Firecrawl instance, falling back to headless Chromium when Firecrawl is unavailable. New items are classified by keyword heuristic by default, or by Claude, an Anthropic model, in batches when the collector's Claude classifier is enabled (it is enabled in production). Either classifier path caps stored confidence at 0.65, so an automated discovery never reads as more certain than an editor's review. Change detections on already-tracked records are different: they store a relevance score of 1 because the score there records certainty that a known instrument's page changed, not classifier confidence — those detections are still editor-gated before anything publishes. Detections are validated and committed. The site reads that data from disk and revalidates hourly; there is no runtime database.
High-confidence detections are staged in data/source-reviews.json; a reviewer uses the local stage → approve → publish workflow before they enter the register. The public policy timeline exposes only verified lifecycle events linked to visible register records. Verified announcements and milestones belong in Developments; legacy examples still stored in data/timeline.json are projected there without duplicating their evidence. The collector never writes to policies.json directly, and CI enforces that.
- Next.js 16 App Router, React 19, TypeScript 5 (strict)
- Tailwind CSS 4, shadcn/ui on Radix UI, D3.js
- Cheerio for scraping; keyword heuristic or Claude (Anthropic) for relevance classification
- Vitest; daily scheduled collection on the maintainer's server (GitHub Actions kept as a manual fallback); self-hosted behind a Cloudflare tunnel
Prerequisites: Node.js >=20.19.0, npm.
npm install
cp .env.example .env.local # optional — the site runs with no keys at all
npm run devOpen http://localhost:3000.
npm run dev # local dev server
npm run build # production build
npm run start # run the production server
npm run lint # ESLint
npm run test # Vitest
npm run validate:data # structural validation of the repo data files
npm run canonicalize:urls # normalize legacy/manual source URL variants
npm run check # lint + strict typecheck + test + validate + build
npm run collect # run one collection pass (add -- --dry-run to preview)
npm run audit:sources # live health check of automatic discovery sources
npm run audit:register # compare curated source fingerprints
npm run mcp # run the local MCP source-ingest serversrc/app/ App Router pages and read-only API routes
src/components/ UI, layout, network, and visualisation components
src/lib/ data service, validation, analysis helpers
src/lib/pipeline/ collector: sources, extract, classify, orchestrate
src/mcp/ local MCP source-ingest server (curated publishing)
src/types/ shared domain types
public/data/ public-safe canonical data served directly as open JSON
data/ editorial register/data, collector state, reviews, coverage
scripts/ collector, source audits, validation and migrations
docs/ operational documentation
content/blog/ MDX blog posts
Everything the site shows is also available as plain JSON, for example:
https://policai.org/data/policies.jsonhttps://policai.org/data/developments.jsonhttps://policai.org/data/timeline.json
Policy, agency, and timeline JSON is served through read-only route handlers so unverified, stale, or withheld editorial records cannot bypass the same public filters as the site.
- Documentation index
- Collector operations guide
- Information trust model
- Architecture
- Scripts overview
- Agent instructions
The site is self-hosted at policai.org, served by next start behind a Cloudflare tunnel. The host pulls main on a timer: data-only commits are picked up by ISR within the hour, and code changes trigger a rebuild and restart. See docs/hosting-argus.md. The collector runs on the same host, in a separate checkout on a daily schedule, and pushes its commits to the same repository. It retrieves browser-only candidate pages through a self-hosted Firecrawl instance and, in production, classifies candidates with Claude through the Claude Code CLI already installed on the host. See docs/collector.md.
See CONTRIBUTING.md. Data corrections are especially welcome: every record links its official source, and npm run validate:data checks structure before CI does.