Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/audit-full.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ jobs:
BASE_PATH: /

- name: Boot web apps + runtime API for the qa-site smoke
env:
# Per-run credential for the ephemeral loopback runtime only. This is
# not a repository or environment secret and grants no external access.
ALLOY_API_KEY: runtime-audit-${{ github.run_id }}-${{ github.run_attempt }}
COMMIT_SHA: ${{ github.sha }}
run: |
set -uo pipefail
# 'serve' is a pinned root devDependency installed by the frozen pnpm
Expand Down Expand Up @@ -158,6 +163,9 @@ jobs:
env:
CI: 'true'
NODE_ENV: test
API_BASE_URL: http://127.0.0.1:8080
# Matches the per-run credential on the loopback runtime boot step.
SMOKE_API_KEY: runtime-audit-${{ github.run_id }}-${{ github.run_attempt }}
# Cap turbo fan-out so the full-repo typecheck (composite tsc builds,
# each allowed --max-old-space-size=6144) does not exhaust the 16 GB
# runner and get OOM-killed mid-run ("The operation was canceled").
Expand Down
47 changes: 26 additions & 21 deletions docs/ops/audit-harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

## Overview

The runtime audit harness provides a single command that boots every product,
runs a comprehensive set of quality checks, captures evidence, and produces a
human-readable summary report. It is the required gate for every change in this
workspace.
The runtime audit harness provides a single command that runs the workspace's
P0 and P1 quality checks, captures evidence, and produces a human-readable
summary report. Its hosted workflow is fail-closed for P0 failures, but it is
not currently a required branch-protection status check.

## Quick start

Expand All @@ -16,7 +16,7 @@ pnpm audit:full
# Fast local iteration (skip install + E2E)
pnpm audit:full:fast

# CI mode — same as fast but exits 2 on P1 failures
# CI mode — skips install + E2E; exits 1 on P0 and records P1 as advisory
pnpm audit:full:ci
```

Expand All @@ -26,26 +26,27 @@ pnpm audit:full:ci
|----------|------|-------------|
| P0 | install | `pnpm install --frozen-lockfile` |
| P0 | typecheck | TypeScript compilation across all packages |
| P0 | lint | ESLint across the full workspace |
| P0 | lint | Biome lint across the full workspace |
| P0 | test | Unit + proof-chain tests |
| P0 | build | Full recursive build of all artifacts |
| P0 | audit:routes | Route registry completeness and classification |
| P0 | qa:site | Public routes + links + trust pages + meta + empty-states |
| P0 | smoke:product-mode | Runtime identity, readiness, API-key rejection, and tenant-scoped read |
| P1 | audit:mocks | Detect real API calls leaking through mock boundaries |
| P1 | audit:copy | UI copy consistency (no placeholder text) |
| P1 | audit:deps | Dependency health (missing/circular/unused) |
| P1 | audit:design-system | Component token compliance |
| P1 | audit:broken-links | Internal hyperlink integrity |
| P1 | qa:a11y | Accessibility audit (axe-core) |
| P1 | brand:check | Brand token and copy compliance |
| P1 | smoke:product-mode | Product-mode isolation smoke tests |
| P1 | docs:claims-check | Documented claims vs. codebase reality |
| P1 | e2e | Playwright end-to-end tests (skipped with `--skip-e2e`) |

**P0 failures are merge-blocking.** A failure in any P0 step aborts the
pipeline immediately and exits with code 1. P1 failures are always recorded
as advisory warnings and never affect the exit code — they appear in the
summary report so they can be tracked and resolved over time.
**P0 failures are harness-blocking.** A failure in any P0 step aborts the
pipeline immediately, exits with code 1, and fails the hosted Runtime Audit
job. P1 failures are always recorded as advisory warnings and never affect the
exit code — they appear in the summary report so they can be tracked and
resolved over time.

## Evidence

Expand All @@ -71,20 +72,24 @@ it in under 60 seconds.

## CI integration

The harness runs as the `Runtime Audit Harness` job in
`.github/workflows/audit-full.yml` on every pull request and push to
`master`/`main`. The job:
The harness runs in the `Runtime Audit (audit:full)` job of the
`Runtime Audit Harness` workflow (`.github/workflows/audit-full.yml`) on pull
requests and pushes to `master`/`main` (pushes changing only
`replit-sync/**` are ignored), and by manual dispatch. The job:

1. Installs dependencies.
2. Runs `pnpm audit:full:ci` (all steps except E2E; exits 1 on P0 failure only).
3. Uploads the entire `artifacts/audit/evidence/` tree as a GitHub Actions
2. Builds the workspace artifacts and boots the local product/runtime targets.
3. Runs `pnpm audit:full:ci` (the harness marks install and E2E skipped; exits 1
on P0 failure only).
4. Uploads the entire `artifacts/audit/evidence/` tree as a GitHub Actions
artifact named `audit-evidence-<run-id>`, retained for 30 days.
4. Prints the `latest/summary.md` to the job log regardless of pass/fail.
5. Prints the `latest/summary.md` to the job log regardless of pass/fail.

P0 failures cause the job to exit non-zero, blocking the merge. P1 failures
are advisory and never block the merge — they appear in the summary report
so they can be tracked and resolved. The existing `ci-gate` job in `ci.yml`
remains unchanged; this workflow is an additive, standalone gate.
P0 failures cause the job to exit non-zero. P1 failures remain advisory and
appear in the summary report. Under the current live branch-protection rules,
Runtime Audit is hosted evidence but is not a required status context, so its
failure does not mechanically prevent a merge. The existing jobs in `ci.yml`
remain unchanged; this workflow is an additive, standalone check.

## Interpreting the summary

Expand Down
17 changes: 9 additions & 8 deletions scripts/audit-full.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@
*
* Priority semantics
* P0 — Blocking: boot failure, typecheck, lint, unit tests, build, broken core
* public routes. Pipeline aborts on first P0 failure. Exit code 1.
* public routes, runtime readiness, and API-key enforcement. Pipeline
* aborts on first P0 failure. Exit code 1.
* P1 — Advisory: mocks, copy, deps, design-system, broken links, a11y, brand,
* docs claims, smoke, E2E. Recorded but do NOT block. Exit code 0.
* docs claims, E2E. Recorded but do NOT block. Exit code 0.
*
* Evidence layout
* artifacts/audit/evidence/<timestamp>/<step>/
Expand Down Expand Up @@ -103,6 +104,12 @@ const STEPS = [
cmd: 'pnpm run qa:site',
priority: 'P0',
},
{
id: 'smoke-product-mode',
label: 'Smoke: product mode',
cmd: 'pnpm run smoke:product-mode',
priority: 'P0',
},
// Advisory checks — failures recorded and reported, but do not block merges.
{
id: 'audit-mocks',
Expand Down Expand Up @@ -146,12 +153,6 @@ const STEPS = [
cmd: 'pnpm run brand:check',
priority: 'P1',
},
{
id: 'smoke-product-mode',
label: 'Smoke: product mode',
cmd: 'pnpm run smoke:product-mode',
priority: 'P1',
},
{
id: 'docs-claims-check',
label: 'Docs: claims check',
Expand Down
Loading
Loading