feat(ci): automated performance regression detection in CI pipeline

.github/workflows/performance-regression.yml

@@ -0,0 +1,105 @@
+name: Performance Regression
+
+on:
+  pull_request:
+    branches: [main, develop]
+    paths:
+      - "src/**"
+      - "performance/**"
+      - "tests/performance/**"
+      - "scripts/analyze-performance.ts"
+      - "scripts/update-performance-baseline.ts"
+      - "playwright.performance.config.ts"
+      - ".github/workflows/performance-regression.yml"
+      - "package.json"
+      - "package-lock.json"
+  push:
+    branches: [main]
+    paths:
+      - "src/**"
+      - "performance/budgets.json"
+      - "tests/performance/**"
+      - "scripts/**"
+      - ".github/workflows/performance-regression.yml"
+  schedule:
+    - cron: "0 4 * * 1"
+  workflow_dispatch:
+
+concurrency:
+  group: performance-regression-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: write
+  pull-requests: write
+
+env:
+  NODE_VERSION: 20
+
+jobs:
+  performance-regression:
+    name: Performance Regression
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install chromium --with-deps
+
+      - name: Build application
+        run: npm run build
+
+      - name: Run performance benchmarks
+        run: npx playwright test --config=playwright.performance.config.ts
+
+      - name: Analyze performance regression
+        id: analyze
+        continue-on-error: true
+        run: npx tsx scripts/analyze-performance.ts
+
+      - name: Upload performance report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: performance-report
+          path: performance-results/
+          retention-days: 30
+
+      - name: Post PR comment
+        if: github.event_name == 'pull_request' && always()
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          if [ ! -f performance-results/performance-comment.md ]; then
+            echo "No performance comment found — skipping."
+            exit 0
+          fi
+          gh pr comment ${{ github.event.pull_request.number }} --body-file performance-results/performance-comment.md || echo "Could not post PR comment (may be a fork PR)."
+
+      - name: Refresh baseline on main
+        if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request' && steps.analyze.outcome == 'success'
+        run: |
+          npx tsx scripts/update-performance-baseline.ts
+          if git diff --quiet performance/baseline.json; then
+            echo "Baseline unchanged — skipping commit."
+          else
+            git config user.name "github-actions[bot]"
+            git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+            git add performance/baseline.json
+            git commit -m "chore(perf): refresh performance baseline [skip ci]"
+            git push
+          fi
+
+      - name: Fail on performance regression
+        if: github.event_name == 'pull_request' && steps.analyze.outcome == 'failure'
+        run: exit 1

.gitignore

@@ -47,3 +47,6 @@ next-env.d.ts
 
 # Lighthouse CI local artifacts
 .lighthouseci/
+
+# performance regression
+/performance-results/

docs/performance-regression-detection.md

@@ -0,0 +1,157 @@
+# Automated Performance Regression Detection in CI
+
+## Problem statement
+
+Performance regressions can silently slip into the frontend: a heavier bundle,
+a slower API handler, or an expensive render path degrades the P99 latency of
+critical paths without breaking any functional test. This document describes
+the automated performance regression detection pipeline that runs in CI to
+catch those regressions before they reach production.
+
+## Goals and technical bounds
+
+- **Performance target:** critical paths must stay below **100ms P99**.
+- **Availability target:** 99.99% uptime — performance gates run on every pull
+  request so regressions never merge.
+- **Scope:** system-wide; the same measurement + baseline + budget mechanism is
+  used by the PR gate, the canary analysis, and the SLO monitoring burn-rate
+  alerts (`docs/slo-monitoring.md`).
+- **Security:** the pipeline only measures HTTP endpoints and browser metrics
+  exposed by the application. No secrets are used; the baseline is a committed,
+  reviewable JSON file.
+
+## Architecture
+
+```
+                    Pull request / main push
+                              │
+                              ▼
+              ┌─────────────────────────────┐
+              │  performance-regression.yml │  (GitHub Actions)
+              └─────────────────────────────┘
+                              │
+          build app · start server · run benchmarks
+                              │
+                              ▼
+        ┌─────────────────────────────────────────────┐
+        │  Playwright benchmark (tests/performance)   │
+        │  measures critical paths: API latency, TTFB │
+        └─────────────────────────────────────────────┘
+                              │  writes raw samples
+                              ▼
+        ┌─────────────────────────────────────────────┐
+        │  scripts/analyze-performance.ts             │
+        │  summarize → compare vs budgets + baseline  │
+        └─────────────────────────────────────────────┘
+                              │
+        ┌─────────────────────┼──────────────────────┐
+        ▼                     ▼                      ▼
+   report.json          report.html            PR comment
+   (machine-           (dashboard for         (human-readable
+    readable)           operators)             findings table)
+        │
+        ▼
+   Gate: exit 1 when a budget is breached or a regression
+   exceeds the baseline tolerance → PR is blocked.
+```
+
+### Components
+
+| Component | Location | Responsibility |
+| --- | --- | --- |
+| Budgets | `performance/budgets.json` | Declarative critical-path definitions with P99 budgets and regression tolerance. |
+| Baseline | `performance/baseline.json` | Committed, reviewable reference measurements refreshed on `main`. |
+| Core logic | `src/utils/performanceRegression.ts` | Pure percentile/summary/regression-detection functions with unit tests. |
+| Benchmark | `tests/performance/performance.spec.ts` | Playwright tests that capture raw latency samples. |
+| Analyzer | `scripts/analyze-performance.ts` | Summarizes samples, evaluates budgets + baseline, writes reports, sets the gate. |
+| Baseline updater | `scripts/update-performance-baseline.ts` | Copies the latest measurements into the committed baseline. |
+| CI workflow | `.github/workflows/performance-regression.yml` | Builds, measures, analyzes, comments on the PR, blocks on regressions. |
+
+## Critical paths and budgets
+
+The budgets file declares every critical path that is measured. Each entry
+carries:
+
+- `id` and `name` — stable identifier and human-readable label.
+- `metric` — the summary metric compared against the budget (`p99`, `p95`, or
+  `mean`). Critical paths use `p99` by default.
+- `budgetMs` — the hard ceiling (e.g. `100` for API critical paths, per the
+  100ms P99 target).
+- `regressionTolerancePercent` — how far the current measurement may drift
+  above the committed baseline before it is flagged as a regression (guards
+  against CI runner noise).
+- `minSampleCount` — minimum number of samples required to make a decision;
+  below this the finding is `insufficient-data` and does not fail the gate.
+
+Current critical paths:
+
+| id | Path | Metric | Budget |
+| --- | --- | --- | ---: |
+| `api-runtime-config-audit` | `/api/runtime-config/audit` | P99 | 100ms |
+| `api-rate-limit` | `/api/rate-limit` | P99 | 100ms |
+| `page-home-ttfb` | `/` | P99 | 1000ms |
+| `page-export-ttfb` | `/export` | P99 | 1000ms |
+| `page-home-lcp` | `/` | P99 | 2500ms |
+| `page-export-lcp` | `/export` | P99 | 2500ms |
+
+## Regression detection rules
+
+For each budget, the analyzer computes percentiles from the raw samples
+(`p50`, `p95`, `p99`, `mean`) and classifies the finding:
+
+| Status | Condition | Gate |
+| --- | --- | --- |
+| `pass` | Measurement within budget and within baseline tolerance | ✅ |
+| `budget-breach` | `current > budgetMs` | ❌ blocks PR |
+| `regression` | `current > baseline × (1 + tolerance)` while still under budget | ❌ blocks PR |
+| `insufficient-data` | Fewer than `minSampleCount` samples collected | ⚠️ does not block |
+
+Regressions are relative to the committed baseline so meaningful slowdowns are
+caught, while tolerance absorbs CI-runner variance. Hard budget breaches always
+fail regardless of the baseline.
+
+## Monitoring, alerting, and dashboards
+
+1. **PR status check** — `Performance Regression / performance` is the alert:
+   a regression or budget breach marks the check red and blocks the merge.
+2. **PR comment** — the workflow posts a markdown table of findings to the PR
+   (`performance-results/performance-comment.md`), so reviewers see exactly
+   which critical path regressed and by how much.
+3. **HTML dashboard** — `performance-results/report.html` is uploaded as a
+   workflow artifact for the run-level breakdown.
+4. **Machine-readable report** — `performance-results/report.json` is consumed
+   by tooling and can be folded into the SLO monitoring panel
+   (`src/utils/slo.ts`), which already targets `< 100ms` P99 for critical
+   paths.
+
+## Blue-green deployment and canary analysis
+
+The same baseline mechanism supports the blue-green rollout described in
+`docs/TRACING_DEPLOYMENT.md`:
+
+1. The **green (candidate)** environment runs the identical benchmark suite.
+2. `scripts/analyze-performance.ts` compares the green measurements against the
+   committed baseline (which reflects the current **blue** environment).
+3. Promotion proceeds only when the canary is within the budget and the
+   baseline tolerance — mirroring the "Critical Path Latency P99 < 100ms"
+   canary threshold in the tracing deployment runbook.
+4. If the canary regresses, the load balancer stays on blue and the incident
+   runbook (`docs/runbooks/performance-regression.md`) applies.
+
+## Baseline lifecycle
+
+- `performance/baseline.json` is committed and reviewed like any other source
+  file.
+- On every push to `main`, the workflow re-measures the critical paths and
+  refreshes the baseline so it tracks the actual runner environment.
+- The baseline can also be refreshed manually:
+
+  ```bash
+  npm run test:performance        # measure critical paths locally
+  npm run performance:update-baseline
+  ```
+
+## Runbook
+
+See `docs/runbooks/performance-regression.md` for triage, remediation, and how
+to update budgets.

docs/runbooks/performance-regression.md

@@ -0,0 +1,106 @@
+# Runbook: Performance Regression Gate
+
+The `Performance Regression` workflow measures critical-path latency on every
+pull request and blocks the merge when a budget is breached or a regression
+exceeds the baseline tolerance. This runbook covers triage, remediation, and
+maintenance.
+
+Architecture and design: [docs/performance-regression-detection.md](../performance-regression-detection.md).
+
+## What the gate does
+
+1. Builds the application and starts the production server.
+2. Runs the Playwright benchmark (`tests/performance/performance.spec.ts`),
+   which samples:
+   - API round-trip latency for `/api/runtime-config/audit` and
+     `/api/rate-limit` (P99 budget: **100ms**).
+   - TTFB and LCP for `/` and `/export`.
+3. `scripts/analyze-performance.ts` summarizes the samples and compares them
+   against `performance/budgets.json` and `performance/baseline.json`.
+4. Findings are posted as a PR comment and uploaded as artifacts
+   (`performance-report`), and the check fails when anything is not green.
+
+## Reading the report
+
+Every finding has a status:
+
+| Status | Meaning | Blocks merge? |
+| --- | --- | --- |
+| `pass` | Within budget and baseline tolerance | No |
+| `budget-breach` | Hard budget exceeded (e.g. P99 > 100ms) | **Yes** |
+| `regression` | Slower than the committed baseline by > tolerance | **Yes** |
+| `insufficient-data` | Fewer than `minSampleCount` samples | No |
+
+Look at the **Δ vs Baseline** column first: a large positive delta means the
+PR introduced a slowdown on that critical path. A `budget-breach` means the
+path is over the absolute SLO target regardless of history.
+
+## Triage
+
+1. Open the `performance-report` artifact from the failed run and open
+   `report.html` (dashboard) or `report.json` (machine-readable).
+2. Identify the failing budget id and metric.
+3. Compare the current value against the budget and baseline.
+
+### Common causes
+
+- **Bundle bloat:** a new dependency or heavier import in a critical route.
+- **API handler slowdown:** new work in an API route (I/O, crypto, parsing).
+- **Render path regression:** heavier client components affecting LCP.
+- **Baseline drift:** CI runner noise or a stale baseline. If the same code
+  re-runs green on a retry, the failure was noise.
+
+## Remediation
+
+1. Optimize the affected path (code-split the component, cache the API
+   response, move work off the critical path).
+2. Re-run the benchmark locally to confirm the improvement:
+
+   ```bash
+   npm run test:performance
+   npm run performance:analyze
+   ```
+
+3. Push and confirm the check is green.
+
+## Updating budgets or tolerances
+
+Budgets are declared in `performance/budgets.json`:
+
+```json
+{
+  "id": "api-runtime-config-audit",
+  "name": "Runtime Config Audit API",
+  "path": "/api/runtime-config/audit",
+  "metric": "p99",
+  "budgetMs": 100,
+  "regressionTolerancePercent": 20,
+  "minSampleCount": 5
+}
+```
+
+- `budgetMs` is the hard SLO ceiling. Do **not** raise it for the 100ms
+  critical-path target without an explicit exception.
+- `regressionTolerancePercent` absorbs runner noise; raise it only when
+  measuring on a new, noisier environment.
+- `minSampleCount` guards the gate against too few samples to be meaningful.
+
+## Refreshing the baseline
+
+The baseline refreshes automatically on pushes to `main` (and the weekly
+schedule) when the run is green. To refresh manually:
+
+```bash
+npm run test:performance
+npm run performance:update-baseline
+```
+
+Commit the updated `performance/baseline.json` in a reviewable PR.
+
+## Escalation
+
+If a critical path exceeds its budget in production, apply the SLO incident
+runbook ([docs/slo-monitoring.md](../slo-monitoring.md)) and the blue-green
+rollback procedure in [docs/TRACING_DEPLOYMENT.md](../TRACING_DEPLOYMENT.md):
+keep the load balancer on blue, investigate the canary measurements, and only
+promote when the canary is within budget and baseline tolerance.