Platform infrastructure: rate limiting, idempotency, observability, auth policies

[rudingma]

Summary

Stand up the cross-cutting platform layer that several deferred items from Story A (#113) and future features depend on. This is NOT a feature ticket — it's a platform workstream. Meant to land as its own coherent PR (or small sequence of PRs), not bolted into feature work.

Why

Multiple Story A review rounds surfaced platform-level concerns (rate limiting, idempotency, metrics, per-user caps) that are architecturally wrong to implement ad-hoc inside a feature ticket. The production-grade pattern is to apply them uniformly at the middleware/interceptor layer, not per-mutation.

Story A already establishes the schema-layer hooks for these (clientMutationId: String! on every mutation input) so this workstream only needs to add the server-side mechanisms — no client or schema changes required per feature.

Scope

Idempotency middleware

• Postgres table idempotency_keys (user_id, key, operation_name, response_json, created_at; composite PK (user_id, key), TTL-indexed on created_at).
• NestJS interceptor or Apollo plugin that runs before mutation resolvers: reads input.clientMutationId, looks up the table, replays cached response if hit, otherwise executes the mutation and caches the response with a 10-minute TTL.
• Periodic cleanup job (or DELETE WHERE created_at < now() - interval '10 minutes') to prune expired rows.
• Works correctly in multi-pod deployments (DB-backed, not in-memory).
• All existing mutations inherit dedup transparently once this lands.

Rate limiting

• @nestjs/throttler wired with Redis backing (NOT in-memory — we run multi-pod, so in-memory is incorrect for shared throttling).
• Default policy: sensible per-user limits on mutations (e.g., 100/min baseline). Tunable per operation via decorators.
• Rejection path surfaces as a typed GraphQL error (or RateLimitedError payload variant once payload-union conventions settle per docs/api-design.md §3).
• Include the Redis instance as part of the infra spec (likely a small CloudNativePG-adjacent Redis deployment or managed service — decide during planning).

Per-user resource caps

• Policy layer (service or interceptor) that enforces "user X cannot own more than N of entity Y" at mutation-entry.
• Story A case: cap on number of households per user as a lead (configurable; reasonable default like 50).
• Applied uniformly once declared — not per-feature ad-hoc.

Prometheus metrics

• @willsoto/nestjs-prometheus integration.
• Default counters / histograms per mutation (request count, latency, error rate) via a decorator or interceptor.
• Grafana dashboard for mutation-level visibility (counter-per-operation, error-rate-per-operation).
• Alert rules for unusual mutation error rates / latency regressions.

Structured logging migration (optional, paired if convenient)

• Migrate from NestJS's default Logger (string-based) to nestjs-pino (structured JSON logs).
• Loki labels become first-class JSON fields instead of logfmt-parsed substrings.
• All existing logger.log(\event=X key=Y`)calls migrate cleanly tologger.log({ event: 'X', key: Y })`.

Consumed by

• Household Foundation: context capture from product-vision brainstorm #111 Household Foundation milestone:
  • Story A (Household Foundation / A: Create a household and become its lead #113) createHousehold: server-side idempotency activates, rate limit applies, per-user household cap applies, structured logs replace logfmt.
  • Story B (Household Foundation / B: Invite a user; invitee joins as member #114) invite flow mutations: same inheritance.
  • Story C (Household Foundation / C: Manage members (promote / demote / remove / leave) #115), E1 (Household Foundation / E1: Grant data model + templates + apply-at-invite #117), E2 (Household Foundation / E2: Grant enforcement + lead-side edit UI #118): same.
• Future Tasks (v1) and beyond: all mutations inherit uniformly.

Dependencies

• Redis deployment (new infra component — check existing Hetzner k3s cluster capacity).
• Postgres schema change for idempotency_keys table (simple migration).
• Decision on Prometheus retention / Grafana dashboard layout (coordinate with existing monitoring setup).

Non-goals

• Application-layer authz beyond per-user resource caps (that's Story E2's grant-enforcement scope).
• WebSocket / subscription rate limiting (no subscriptions yet).
• Distributed tracing (future workstream if needed).

Acceptance

• Every mutation in the app (current + future) gets idempotency, rate limiting, resource caps, and metrics for free, by decorator or by default.
• No feature ticket needs to re-implement these concerns.
• Loadgen / chaos test verifies idempotent retry works end-to-end across pods.

Follow-up from Story A (#113)

• clientMutationId echo in mutation payloads. Decide whether the idempotency middleware requires each mutation response to echo its clientMutationId back so clients can correlate late responses. If yes: add clientMutationId: String! to CreateHouseholdPayload and every future payload; update docs/api-design.md §3 with the echo convention. Folded here from Story A because the decision depends on this workstream's design.