diff --git a/.claude/development/smoke-prod-guide.md b/.claude/development/smoke-prod-guide.md index 3820ef058..cbfb9bb13 100644 --- a/.claude/development/smoke-prod-guide.md +++ b/.claude/development/smoke-prod-guide.md @@ -22,7 +22,7 @@ with 2s backoff per call. Exit codes: `0` ok, `1` fail, `2` skipped. | ID | What it checks | Trigger | |----|----------------|---------| -| `health` | `GET $SUPABASE_URL/functions/v1/health` returns 200 | always-on canary | +| `health` | `GET $SUPABASE_URL/functions/v1/health` returns 200; `verbose=true` body's `redis` check must be `healthy` | always-on canary | | `website-device-page` | `GET https://www.skillsmith.app/device` returns 200 + contains `data-smoke="device-input"` | `packages/website/src/pages/device.astro` change | | `edge-fn-auth-device` | `auth-device-code` POST → 200/400 (NOT 404); `auth-device-preview` GET (no JWT) → 401 | `supabase/functions/auth-device-{code,preview,approve,token}/**` change | | `edge-fn-skills-search` | Authenticated GET with `X-API-Key: $SMOKE_SKILLS_API_KEY` → 200; asserts `user_api_usage.search_count` incremented by 1 | `supabase/functions/skills-search/**` or `_shared/usage-counter.ts` / `_shared/auth-middleware.ts` change | @@ -107,6 +107,11 @@ Triage steps: or hotfix. - Assertion is wrong (smoke-harness bug) → fix the check function; deploy is fine. + - `redis` check present but not `healthy` → real Upstash issue (bad/rotated + token, network, quota); see docs/internal/runbooks/upstash-redis-operations.md. + Not a full outage by itself — rate-limiter/quota-enforcer/tier-cache/ + circuit-breaker all degrade gracefully, but treat as urgent (rotation + may be needed). ## Skipping smoke for a PR diff --git a/.env.schema b/.env.schema index 12bc93ddf..339bf855c 100644 --- a/.env.schema +++ b/.env.schema @@ -438,6 +438,41 @@ GA4_MEASUREMENT_ID= # @type=string @required=false @sensitive GA4_API_SECRET= +# ============================================================================= +# UPSTASH REDIS (RATE LIMITING / CACHING) — SMI-1231 +# ============================================================================= + +# Upstash Redis REST URL - Backs rate-limiter.ts (sliding-window rate limits), +# quota-enforcer.ts and tier-cache.ts (look-aside caches in front of Postgres +# RPCs, self-healing on failure), and circuit-breaker.ts (distributed trip +# state) — one instance, four consumers. All four fail open to an in-memory, +# per-isolate fallback if this is unset/unreachable. rate-limiter.ts is the +# one module where that matters: a per-isolate in-memory counter isn't +# shared across concurrent Supabase edge instances, so a silent outage +# measurably weakens (not disables) per-tier rate enforcement. Full +# architecture + rotation: docs/internal/runbooks/upstash-redis-operations.md +# Get from: Upstash Console → Redis → (database) → REST API → UPSTASH_REDIS_REST_URL +# Staging may omit this entirely (graceful degradation) — see +# docs/internal/implementation/supabase-staging-environment.md +# @type=string(startsWith=https://) @required=false @sensitive=false +UPSTASH_REDIS_REST_URL= + +# Upstash Redis REST Token - Paired with UPSTASH_REDIS_REST_URL above. MUST +# be a read+write token: rate-limiter increments counters, quota-enforcer and +# tier-cache write cache entries, circuit-breaker writes trip state. Upstash's +# read-only token option is NOT viable here — do not scope this down in a +# future least-privilege pass (see upstash-redis-operations.md). +# Get from: Upstash Console → Redis → (database) → REST API → UPSTASH_REDIS_REST_TOKEN +# @type=string @required=false @sensitive +UPSTASH_REDIS_REST_TOKEN= + +# Rate Limit Fail-Closed Mode - When "true", checkRateLimit() in +# rate-limiter.ts rejects ALL requests if Redis is unreachable/misconfigured, +# instead of falling back to an in-memory limiter. Default (unset/"false") +# is fail-open. Registered guard: docs/internal/process/guards-and-opt-outs.md. +# @type=enum(true,false) @required=false @sensitive=false +RATE_LIMIT_FAIL_CLOSED= + # ============================================================================= # CLOUDINARY MEDIA MANAGEMENT # ============================================================================= diff --git a/CLAUDE.md b/CLAUDE.md index 7d31e41b0..851354201 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,6 +28,7 @@ Detailed guides extracted via progressive disclosure. CLAUDE.md contains essenti | [edge-function-attribution-queries.md](.claude/development/edge-function-attribution-queries.md) | Canonical pooler queries for edge function attribution monitoring (SMI-4370 / Wave 4d) | | [concurrency-patterns.md](.claude/development/concurrency-patterns.md) | Pattern-to-incident-to-canonical-fix index for the five `concurrency-auditor` patterns (SMI-4895/4896/4861/4887) | | [guards-and-opt-outs.md](docs/internal/process/guards-and-opt-outs.md) | Canonical Guards & Opt-Outs registry (guard → trigger → marker → disable var → scope) — SMI-5418 DoD #5 | +| [upstash-redis-operations.md](docs/internal/runbooks/upstash-redis-operations.md) | Upstash Redis architecture (4 consumers), setup, credential rotation, health-check monitoring | **Implementation plan template**: [.claude/templates/implementation-plan.md](.claude/templates/implementation-plan.md) — use this structure for all plans in `docs/internal/implementation/`. diff --git a/docs/internal b/docs/internal index 0c741232b..82b4d22d2 160000 --- a/docs/internal +++ b/docs/internal @@ -1 +1 @@ -Subproject commit 0c741232b2352b8ed062e15d020fb57f9b664da3 +Subproject commit 82b4d22d28761b801ff96ff216c6f7d752d5728f diff --git a/supabase/functions/_shared/health-checks.helpers.test.ts b/supabase/functions/_shared/health-checks.helpers.test.ts index 3a8a2aa9a..579104008 100644 Binary files a/supabase/functions/_shared/health-checks.helpers.test.ts and b/supabase/functions/_shared/health-checks.helpers.test.ts differ diff --git a/supabase/functions/_shared/health-checks.probes.test.ts b/supabase/functions/_shared/health-checks.probes.test.ts index b87a4f2bb..c0e6c2d97 100644 Binary files a/supabase/functions/_shared/health-checks.probes.test.ts and b/supabase/functions/_shared/health-checks.probes.test.ts differ diff --git a/supabase/functions/_shared/health-checks.probes.ts b/supabase/functions/_shared/health-checks.probes.ts index 0515b065f..1e61f6ac2 100644 Binary files a/supabase/functions/_shared/health-checks.probes.ts and b/supabase/functions/_shared/health-checks.probes.ts differ diff --git a/supabase/functions/_shared/health-checks.readers.test.ts b/supabase/functions/_shared/health-checks.readers.test.ts index d02687d2c..b51fb7c25 100644 Binary files a/supabase/functions/_shared/health-checks.readers.test.ts and b/supabase/functions/_shared/health-checks.readers.test.ts differ diff --git a/supabase/functions/_shared/health-checks.ts b/supabase/functions/_shared/health-checks.ts index 5a62ee954..25ae02100 100644 Binary files a/supabase/functions/_shared/health-checks.ts and b/supabase/functions/_shared/health-checks.ts differ diff --git a/supabase/functions/_shared/rate-limiter.health.test.ts b/supabase/functions/_shared/rate-limiter.health.test.ts new file mode 100644 index 000000000..00e83a374 Binary files /dev/null and b/supabase/functions/_shared/rate-limiter.health.test.ts differ diff --git a/supabase/functions/_shared/rate-limiter.health.ts b/supabase/functions/_shared/rate-limiter.health.ts new file mode 100644 index 000000000..52ac3dd5c Binary files /dev/null and b/supabase/functions/_shared/rate-limiter.health.ts differ diff --git a/supabase/functions/_shared/rate-limiter.ts b/supabase/functions/_shared/rate-limiter.ts index 6ccf35d01..82dc0d4ed 100644 Binary files a/supabase/functions/_shared/rate-limiter.ts and b/supabase/functions/_shared/rate-limiter.ts differ diff --git a/supabase/functions/health/index.test.ts b/supabase/functions/health/index.test.ts index fc408f633..c76464f75 100644 Binary files a/supabase/functions/health/index.test.ts and b/supabase/functions/health/index.test.ts differ diff --git a/supabase/functions/health/index.ts b/supabase/functions/health/index.ts index daf8e3365..2132828aa 100644 Binary files a/supabase/functions/health/index.ts and b/supabase/functions/health/index.ts differ diff --git a/supabase/functions/status-check/index.test.ts b/supabase/functions/status-check/index.test.ts index 380726ed5..7302f4f11 100644 Binary files a/supabase/functions/status-check/index.test.ts and b/supabase/functions/status-check/index.test.ts differ diff --git a/supabase/functions/status-check/index.ts b/supabase/functions/status-check/index.ts index ae1c426a3..50f4145fb 100644 Binary files a/supabase/functions/status-check/index.ts and b/supabase/functions/status-check/index.ts differ diff --git a/supabase/migrations/20260719030000_status_component_rate_limiting.sql b/supabase/migrations/20260719030000_status_component_rate_limiting.sql new file mode 100644 index 000000000..0b1069f01 Binary files /dev/null and b/supabase/migrations/20260719030000_status_component_rate_limiting.sql differ diff --git a/supabase/rollbacks/20260719030000_status_component_rate_limiting_down.sql b/supabase/rollbacks/20260719030000_status_component_rate_limiting_down.sql new file mode 100644 index 000000000..2ffc54f9e --- /dev/null +++ b/supabase/rollbacks/20260719030000_status_component_rate_limiting_down.sql @@ -0,0 +1,13 @@ +-- Rollback for 20260719030000_status_component_rate_limiting.sql +-- SMI-5768 +-- +-- Removes the 'rate-limiting' status_components row. Cascades (ON DELETE +-- CASCADE, migration 20260719010000_status_page_schema.sql) to any +-- status_checks/status_daily_rollups rows already accumulated for this +-- component — that history is permanently lost, same tradeoff as the other +-- 6 components' rollback would carry. +-- +-- Safe to run standalone via: +-- docker exec skillsmith-dev-1 varlock run -- ./scripts/pooler-psql.sh -f supabase/rollbacks/20260719030000_status_component_rate_limiting_down.sql + +DELETE FROM status_components WHERE slug = 'rate-limiting';