Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 0 additions & 122 deletions .github/workflows/collect.yml

This file was deleted.

4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Policai is an Australian AI policy tracker. It maintains a curated register of AI policy, regulation, governance and court guidance across federal and state/territory jurisdictions, plus an automated "developments" feed of newly detected policy activity.

**Git is the database.** All canonical data is JSON committed to this repository (`public/data/`, `data/`). The deployed site only reads that data. It is self-hosted at [policai.org](https://policai.org) behind a Cloudflare tunnel. Collection runs daily on the maintainer's server, in a checkout separate from the one that serves the site, and pushes new detections to GitHub; the serving checkout pulls those commits and serves them through ISR without a rebuild. GitHub Actions no longer schedules collection; `.github/workflows/collect.yml` is kept as a manual fallback. There is no runtime database, no auth, and no admin dashboard.
**Git is the database.** All canonical data is JSON committed to this repository (`public/data/`, `data/`). The deployed site only reads that data. It is self-hosted at [policai.org](https://policai.org) behind a Cloudflare tunnel. Collection runs daily on the maintainer's server, in a checkout separate from the one that serves the site, and pushes new detections to GitHub; the serving checkout pulls those commits and serves them through ISR without a rebuild. There is no runtime database, no auth, and no admin dashboard.

## Tech Stack

Expand All @@ -18,7 +18,7 @@ Policai is an Australian AI policy tracker. It maintains a curated register of A
- **Analysis:** keyword heuristic by default; Claude, an Anthropic model, batched through the Claude Code CLI on the collection host when `USE_CLAUDE_CLASSIFIER` is set — both paths cap stored/displayed confidence at 0.65 (`MACHINE_CONFIDENCE_CAP`)
- **Scraping:** Cheerio
- **Testing:** Vitest (+ Testing Library)
- **Automation:** a daily scheduled run on the maintainer's server; GitHub Actions (`.github/workflows/collect.yml`) kept as a manual fallback
- **Automation:** a daily scheduled run on the maintainer's server

## Commands

Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Policai

[![Daily collection](https://github.com/l0cka/policai/actions/workflows/collect.yml/badge.svg)](https://github.com/l0cka/policai/actions/workflows/collect.yml)

Policai is an Australian AI policy tracker. It maintains a curated register of AI policy, regulation, governance and court guidance across federal and state/territory jurisdictions, and automatically detects new developments from official government sources every day.

Expand Down Expand Up @@ -28,7 +27,7 @@ Product surface:
- `data/source-reviews.json` — detections staged for curated review
- `data/source-monitoring.json` — the manual-source review ledger

The maintainer's server runs the collector daily, from its own checkout, over the official sources that reliably permit machine retrieval. The GitHub Actions workflow ([collect.yml](.github/workflows/collect.yml)) no longer schedules collection and is kept as a manual fallback. Sources protected by browser challenges are kept in the same source catalogue but reviewed through the manual coverage ledger. Candidate pages from browser-only sources are retrieved through a self-hosted Firecrawl instance, falling back to headless Chromium when Firecrawl is unavailable. New items are classified by keyword heuristic by default, or by Claude, an Anthropic model, in batches when the collector's Claude classifier is enabled (it is enabled in production). Either path caps stored confidence at 0.65, so an automated detection never reads as more certain than an editor's review. Detections are validated and committed. The site reads that data from disk and revalidates hourly; there is no runtime database.
The maintainer's server runs the collector daily, from its own checkout, over the official sources that reliably permit machine retrieval. Sources protected by browser challenges are kept in the same source catalogue but reviewed through the manual coverage ledger. Candidate pages from browser-only sources are retrieved through a self-hosted Firecrawl instance, falling back to headless Chromium when Firecrawl is unavailable. New items are classified by keyword heuristic by default, or by Claude, an Anthropic model, in batches when the collector's Claude classifier is enabled (it is enabled in production). Either path caps stored confidence at 0.65, so an automated detection never reads as more certain than an editor's review. Detections are validated and committed. The site reads that data from disk and revalidates hourly; there is no runtime database.

High-confidence detections are staged in `data/source-reviews.json`; a reviewer uses the local stage → approve → publish workflow before they enter the register. Public register and timeline reads only expose verified records. The collector never writes to `policies.json` directly, and CI enforces that.

Expand Down
17 changes: 11 additions & 6 deletions docs/collector.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,11 +228,11 @@ the site. Each run:
5. If collection reported failed coverage, fail the run only after that
operational state is preserved

[`.github/workflows/collect.yml`](../.github/workflows/collect.yml) no longer
schedules collection; it is kept as a manual fallback
(Actions → "Collect AI policy developments" → Run workflow, optionally with a
single source id) and runs the same steps, including opening or commenting on
an issue labelled `collector-failure` on any failure.
The GitHub Actions collection workflow has been retired after the scheduled
run on the maintainer's server produced a clean cycle. Failures still open or
comment on an issue labelled `collector-failure`, now raised by the scheduled
run itself. A manual pass can be run from any checkout with
`npm run collect -- --source=<id>`.

The site's own checkout pulls the push on its timer. Because pages read the
JSON from disk at request time and revalidate hourly, the new data appears
Expand All @@ -245,7 +245,12 @@ version cannot be lost merely because its state write completed first.

**Repository configuration:**

- `COLLECTOR_DEPLOY_KEY` secret — private half of the repo's write deploy key ("collector (collect.yml push)"), used only by the GitHub Actions fallback run. Checkout uses it (`ssh-key:`) so the push authenticates as the deploy key, and the "Protect main" ruleset lists **Deploy keys** as a bypass actor (`bypass_mode: always`). Without this pair the fallback push is rejected with `GH013: Repository rule violations` — the default `GITHUB_TOKEN` cannot be a bypass actor on a user-owned repo. The scheduled collection run pushes over its own SSH-authenticated git remote and does not use this secret. The safety story does not depend on the ruleset here: the registry-guard step runs on both paths and enforces that automation never touches `policies.json`.
- The retired Actions workflow used a write deploy key (`COLLECTOR_DEPLOY_KEY`
secret) as a ruleset bypass actor; both the key and the secret can be
deleted now that the workflow is gone. The scheduled collection run pushes
over its own SSH-authenticated git remote. The safety story does not depend
on the ruleset: the registry guard enforces that automation never touches
`policies.json`.

## Reviewing detections into the register

Expand Down
Loading