release: v1.0.0-alpha.21

.planning/phases/13-launch-verification/evidence/02-deploy.md

@@ -100,8 +100,161 @@ Both release PRs are open and mergeable ([#46](https://github.com/cativo23/clari
 
 ## Task 2: Go/no-go checkpoint
 
-Pending — awaiting Carlos's decision. See the CHECKPOINT REACHED report returned by this executor for the decision context, options, and undo-cost summary.
+**Carlos said "go."** Proceeding with Task 3.
 
 ## Task 3: Tracer (production leg)
 
-Not started — blocked on Task 2's "go" signal. Will be appended to this file on `develop` after Task 2 resolves, per this plan's action step 8.
+### 1. Merge PRs
+
+```bash
+$ gh pr merge 47 --merge --admin   # release/v1.0.0-alpha.20 -> develop
+mergedAt: 2026-09-28T22:46:19Z, mergeCommit: 11fecefa91e863921e81617f74bc9ad7741584bf
+
+$ gh pr merge 46 --merge --admin   # release/v1.0.0-alpha.20 -> main
+mergedAt: 2026-09-28T22:46:27Z, mergeCommit: e248cc7e4733f043259546d24102bcbd8923396a
+```
+**Result:** PASS — both PRs merged, develop first then main, per the plan's step ordering.
+
+### 2. Auto Release
+
+```bash
+$ gh run list -R cativo23/clarify --workflow=auto-release.yml --limit 1
+36494349485  release/v1.0.0-alpha.20  in_progress
+$ gh run watch 36494349485 --exit-status
+✓ release in 6s (ID 109170284474)
+  ✓ Set up job / Checkout / Extract and validate version / Check tag does not already exist
+  ✓ Parse release notes from CHANGELOG.md / Create GitHub Release
+```
+
+```bash
+$ gh release view v1.0.0-alpha.20 -R cativo23/clarify --json tagName,isPrerelease,body
+{"tagName":"v1.0.0-alpha.20","isPrerelease":true,"body":"### Features\n...### Deploy\n..."}
+```
+**Result:** PASS — auto-release.yml created the release itself as a prerelease, notes match the CHANGELOG `[1.0.0-alpha.20]` section exactly. CLAUDE.md's manual tag/`gh release create` steps (4-5) were **not** run — not needed, avoiding a duplicate release.
+
+### 3. CI/CD — build
+
+```bash
+$ gh run list -R cativo23/clarify --workflow=ci-cd.yml --limit 1
+36494363136  v1.0.0-alpha.20  release event, in_progress
+```
+
+**First attempt (run 36494363136):**
+- `build` job: **success** in 3m23s — image built and pushed to Docker Hub (`cativo23/clarify:v1.0.0-alpha.20`, `:latest`, `:sha-*`).
+- `Deploy to Home Server` job: **failed** at step "Copy compose file to server" (`appleboy/scp-action`). Log tail:
+  ```
+  remote server os type is unix
+  scp file to server.
+  2026/09/28 22:54:42 error copy file to dest: ***, error message: dial tcp ***:***: connect: connection timed out
+  ```
+  The first connection (host-type probe) succeeded; the second connection (the actual file transfer) timed out after ~2m15s.
+
+**Diagnosis attempt 1 (before retrying):** direct SSH from this machine to the server succeeded immediately (`ssh -p 52222 cativo23@cativo.dev` → `SSH_OK`), confirming the `SSH_HOST`/`SSH_PORT`/`SSH_USERNAME`/`SSH_PRIVATE_KEY` secrets are all correct and the server is reachable. This did not look like a credential problem, so per the D-05 fix loop this executor ran ONE rerun of the failed job: `gh run rerun 36494363136 --failed`.
+
+**Second attempt (same run, rerun):**
+- `build` job: **success** again in 3m23s (rebuilt/re-pushed the same tags).
+- `Deploy to Home Server` job: **failed again**, identical symptom:
+  ```
+  remote server os type is unix
+  scp file to server.
+  2026/09/28 23:00:21 error copy file to dest: ***, error message: dial tcp ***:***: connect: connection timed out
+  ```
+  Same ~2m15s timeout on the second connection, same step.
+
+**Result:** **FAIL — two identical failures.** Per this plan's executor notes ("if the same approach fails twice, stop and report"), no third rerun was attempted.
+
+**Additional diagnosis performed (read-only, no server changes):**
+```bash
+$ ssh -p 52222 cativo23@cativo.dev 'sudo -n fail2ban-client status sshd'
+Status for the jail: sshd
+|- Currently banned: 0
+`- Banned IP list:  (empty)
+
+$ ssh -p 52222 cativo23@cativo.dev 'sudo -n grep "2026-09-28 2[23]:" /var/log/fail2ban.log'
+(no output — no ban/unban activity during the failure window 22:50-23:01 UTC)
+
+$ ssh -p 52222 cativo23@cativo.dev 'grep -E "MaxStartups|MaxSessions" /etc/ssh/sshd_config'
+MaxSessions 50
+MaxStartups 20:50:100
+```
+fail2ban is not blocking anything and sshd's connection limits are generous — this rules out a server-side auto-ban or connection-limit cause. `journalctl -u sshd` was not readable by the `cativo23` user (not in `adm`/`systemd-journal` groups), so the exact TCP-level cause on the GitHub Actions → home-server path (ISP/router NAT, ephemeral runner IP, or a transient network condition specific to establishing a *second* connection within the same job) could not be pinned down further from this session.
+
+**Note on the deploy job's actual effect:** because the failure is at "Copy compose file to server" (before "Deploy to server", which was `skipped`), **no `docker compose` command ever ran on the server**. Production is unchanged from its pre-13-03 state (no clarify containers running, per 13-01/13-02's pre-flight evidence) — this is not a partial/broken deploy, it is simply not deployed yet.
+
+**Halting Task 3 here per plan/executor-notes instructions.** The release is published and the image is built and available on Docker Hub; only the SSH-based compose-file copy step is blocked. Steps 4-8 of Task 3 (server health check, HTTPS/TLS checks, image-digest comparison, release-branch cleanup, and this file's own completion) are **not yet done** and depend on a successful deploy job run.
+
+**Recommended next steps for Carlos (not acted on by this executor — outside the D-05 categories and outside D-02's automated-only deploy scope):**
+1. Simplest: just retry again later (`gh run rerun 36494363136 --failed` from `gh` CLI) — the symptom (first connection OK, second connection times out) is consistent with a transient network condition (home router NAT/conntrack table, ISP path, or the specific GitHub Actions runner IP for that job) rather than a fixed misconfiguration, since direct SSH from this session worked both times without issue.
+2. If it keeps failing identically, check the home router/firewall logs for dropped connections around the failure timestamps (this session could not reach beyond `fail2ban` and `sshd_config`, both clean).
+3. A one-time manual `docker compose` deploy via this session's already-working SSH access was considered and intentionally **not** done — 13-CONTEXT.md's D-02 explicitly scopes this phase to the *existing automated* GitHub Actions pipeline ("No new deploy mechanism needed"), and 13-RESEARCH.md flags manual deploy as "not the intended flow" for this phase. Switching to it would be a deviation from an explicit locked decision, not a same-approach retry — flagging for Carlos's call rather than doing it unilaterally.
+
+---
+
+## Orchestrator continuation: root cause found and resolved
+
+The executor's diagnosis (transient network condition) was investigated further by the orchestrator after Carlos reported the deploy as failed. Two more identical failures occurred on manual reruns (3 total), ruling out "transient."
+
+### Actual root cause
+
+`clarify`'s GitHub repo has a **`prod` environment** with its own environment-scoped secrets named `SSH_HOST` and `SSH_PORT` — these existed **before** this phase's work and were never surfaced by 13-02's research, which only checked repo-level secrets (`gh secret list`, which does not include environment-scoped ones). Since `ci-cd.yml`'s `deploy` job declares `environment: prod`, GitHub resolves `secrets.SSH_HOST`/`secrets.SSH_PORT` from the **environment-level** secrets, which silently shadow any repo-level secrets of the same name. The orchestrator's earlier `gh secret set SSH_HOST`/`gh secret set SSH_PORT` (during 13-02's checkpoint continuation) had set **repo-level** secrets, which were never actually used by this job — explaining three consistent, non-transient failures.
+
+**Found by comparing against a working sibling project:** `portfolio-api` (43 production releases to the same host, `environment: prod`) uses `vars.DEPLOY_HOST`/`vars.DEPLOY_PORT` (plain repo variables, not secrets) set to the server's **raw IP** (`167.235.52.161`), sidestepping this exact class of shadowing issue entirely.
+
+**Fix applied:**
+```bash
+$ gh secret set SSH_HOST --env prod -R cativo23/clarify -b "167.235.52.161"
+$ gh secret set SSH_PORT --env prod -R cativo23/clarify -b "52222"
+```
+Both environment-scoped secrets confirmed updated (`updated_at: 2026-09-29T00:59:2Xz`).
+
+### CI/CD — deploy (successful rerun)
+
+```bash
+$ gh run rerun 36494363136 -R cativo23/clarify --failed
+$ gh run view 36494363136 -R cativo23/clarify --json conclusion --jq '.conclusion'
+success
+```
+**Result:** PASS — `Deploy to Home Server` completed. `docker compose -f docker-compose.prod.yml pull/down/up -d` ran on polaris2 for the first time.
+
+### Post-deploy incident: two further env-only bugs, fixed live (no new release needed)
+
+1. **Redis auth vs. TLS conflict.** `/` returned HTTP 500. App logs: `[SECURITY] Redis authentication not configured in production` → `Redis authentication required in production environment` (from `server/utils/rate-limit.ts`'s hard production gate). The compose file's bundled `redis:7-alpine` had no password. The orchestrator initially set a fresh `REDIS_TOKEN` + `--requirepass` on the bundled local Redis, but `rate-limit.ts`/`queue.ts` unconditionally enable TLS (`redisConfig.tls = {}`) whenever `redisToken` is set — code written for Upstash, and the bundled local Redis doesn't speak TLS, so this produced `ETIMEDOUT` on every connection instead. **Resolved per Carlos's direction:** he already runs the same Upstash Redis instance for local dev; the orchestrator had Carlos copy his local `.env`'s `REDIS_HOST`/`REDIS_PORT`/`REDIS_TOKEN` (Upstash) onto the server `.env` in place of the bundled-Redis token, matching the architecture `server/utils/rate-limit.ts` and `queue.ts` actually expect. `docker compose up -d` recreated containers; `/api/health` confirmed `"redis":"connected"` with no auth error.
+   - **Deviation note:** the bundled `redis:7-alpine` container in `docker-compose.prod.yml` is now unused dead weight (still started as a `worker` health dependency) since the app talks to Upstash instead. Not removed in this session — flagged as cleanup for a future phase/plan, not blocking LAUNCH-01..04.
+2. **`SUPABASE_ANON_KEY` / `SUPABASE_KEY` name mismatch.** `/` still 500'd after the Redis fix: `"Your project's URL and Key are required to create a Supabase client!"`. `docker-compose.prod.yml` reads `${SUPABASE_ANON_KEY}` (and logged a compose warning that it was unset) while the server `.env` (copied from Carlos's local dev `.env`) uses the name `SUPABASE_KEY` for the same anon/publishable key. Fixed by adding a `SUPABASE_ANON_KEY` line to the server `.env` duplicating `SUPABASE_KEY`'s value (a public/publishable key, `sb_publishable_...` prefix — safe to duplicate under two names). `docker compose up -d` recreated containers again.
+   - **Deviation note:** this is a pre-existing naming inconsistency between `.env.example`/`docker-compose.prod.yml` (which use `SUPABASE_ANON_KEY`) and the project's actual `.env` convention (`SUPABASE_KEY`, matching `CLAUDE.md`'s documented required-env list). Worth reconciling in a future cleanup plan so a fresh `.env` built from `.env.example` doesn't hit this same gap.
+
+### Final verification (all must_haves)
+
+```bash
+$ curl -s -o /dev/null -w "%{http_code}" https://clarify.cativo.dev/api/health
+200
+$ curl -s https://clarify.cativo.dev/api/health
+{"status":"ok","services":{"database":"unknown","redis":"connected","ai":"active"},"timestamp":"2026-09-29T01:20:28.373Z"}
+$ curl -s -o /dev/null -w "%{http_code}" https://clarify.cativo.dev/
+200
+$ curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}" http://clarify.cativo.dev/
+301 -> https://clarify.cativo.dev/
+$ echo | openssl s_client -connect clarify.cativo.dev:443 -servername clarify.cativo.dev 2>/dev/null | openssl x509 -noout -issuer -dates
+issuer=C=US, O=Let's Encrypt, CN=YR2
+notBefore=Sep 29 00:02:00 2026 GMT
+notAfter=Dec 28 00:01:59 2026 GMT
+$ curl -sI https://clarify.cativo.dev/ | grep -i strict-transport
+strict-transport-security: max-age=31536000; includeSubDomains; preload
+$ ssh -p 52222 cativo23@cativo.dev "docker ps --filter name=clarify --format '{{.Names}}: {{.Status}}'"
+clarify-worker-prod: Up (healthy)
+clarify-redis-prod: Up (healthy)
+clarify-app-prod: Up (healthy)
+$ ssh -p 52222 cativo23@cativo.dev "docker inspect cativo23/clarify:latest --format '{{index .RepoDigests 0}}'"
+cativo23/clarify@sha256:3536b201954146f9d2fb7787fdd1e58abb997ebc16c22ac15383b7b284bee6bc
+$ docker buildx imagetools inspect cativo23/clarify:v1.0.0-alpha.20 --format '{{json .Manifest}}' | jq -r '.digest'
+sha256:3536b201954146f9d2fb7787fdd1e58abb997ebc16c22ac15383b7b284bee6bc
+```
+**Result:** PASS — all Task 3 `must_haves` satisfied. Running image digest matches the published `v1.0.0-alpha.20` tag exactly. Redis reports connected (via Upstash, not the unused bundled container). Certificate is a valid Let's Encrypt cert with ~90 days validity. HSTS present. All three containers healthy.
+
+### Incident note: accidental secret exposure
+
+While debugging the corrupted `.env` (see below), the orchestrator ran `cat .env` over SSH, which printed the full production `.env` — including the real Supabase service key, OpenAI API key, and Stripe test-mode keys — into this session's transcript. Carlos was notified immediately and chose to handle rotation of those credentials himself, separately, rather than block the deploy on it. Flagged here for the record; no further action taken on it in this plan.
+
+**`.env` corruption/recovery (unrelated to the above):** an earlier `sed` append (setting the bundled-Redis `REDIS_TOKEN`, since superseded by the Upstash fix) ran against a `.env` whose last line had no trailing newline, merging `BASE_URL`'s value with the new `REDIS_TOKEN=` line into one corrupted line. Caught immediately (before any container restart used it) and fixed by splitting the merged line back into two proper lines.
+
+**Task 3 status: COMPLETE.** Release published, deployed, and independently re-verified end-to-end in production.

.planning/phases/13-launch-verification/evidence/03-hotfix-alpha21.md

@@ -0,0 +1,28 @@
+# Hotfix Release v1.0.0-alpha.21
+
+**Date:** 2026-09-29
+**Trigger:** After v1.0.0-alpha.20's first production deploy, Carlos reported the dark mode toggle
+broken plus console errors: two CSP `script-src` violations and
+`Uncaught TypeError: Cannot read properties of undefined (reading 'app')`.
+
+## Root cause
+
+`nuxt.config.ts`'s custom `security.headers.contentSecurityPolicy["script-src"]` override (added to
+allow Stripe/Cloudflare domains) omitted the literal placeholder string `'nonce-{{nonce}}'` that
+`nuxt-security`'s `50-updateCsp.js` Nitro plugin looks for verbatim to substitute the real per-request
+nonce into the response CSP header. Confirmed live: the header's `script-src` carried zero
+`'nonce-*'` sources across repeated requests to `https://clarify.cativo.dev/`, while the HTML's own
+`nonce="..."` attribute on every `<script>` tag correctly rotated per request — proving the
+substitution never ran. Every inline script, including Nuxt's own `__NUXT_DATA__` hydration payload,
+was blocked by the browser, breaking client-side hydration (and with it, any client-only reactivity
+such as the dark mode toggle).
+
+## Fix
+
+PR #48 (`fix/csp-nonce-script-src` → `develop`): added `"'nonce-{{nonce}}'"` back into the `script-src`
+array in `nuxt.config.ts`.
+
+## This release
+
+Same-day hotfix following v1.0.0-alpha.20's tracer deploy. Repeats the same GitFlow release process
+(D-02) verified working in plan 13-03.

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.0.0-alpha.21] - 2026-09-29
+
+### Fixed
+- Restored `nuxt-security`'s `'nonce-{{nonce}}'` placeholder in the custom `script-src` CSP directive. Its absence silently disabled nonce substitution in the response header, so every inline `<script>` on the page — including Nuxt's own hydration payload — was blocked by CSP in production. This broke client-side hydration entirely (dark mode toggle and other client reactivity silently no-op'd, with `Cannot read properties of undefined (reading 'app')` in the console).
+
 ## [1.0.0-alpha.20] - 2026-09-28
 
 ### Features

nuxt.config.ts

@@ -25,6 +25,14 @@ export default defineNuxtConfig({
           "'self'",
           "https://js.stripe.com",
           "https://challenges.cloudflare.com",
+          // nuxt-security's per-request nonce (see 50-updateCsp.js) only gets
+          // substituted into the CSP header when this exact placeholder is
+          // present in the directive array. Omitting it (as this override
+          // previously did) silently drops the nonce from the response
+          // header even though it's still stamped onto every <script> tag,
+          // which blocks all inline scripts — including Nuxt's own payload
+          // hydration script — under CSP.
+          "'nonce-{{nonce}}'",
           // Allow unsafe scripts ONLY in development for HMR and hydration
           ...(process.env.NODE_ENV !== "production"
             ? ["'unsafe-inline'", "'unsafe-eval'"]

package.json

@@ -1,6 +1,6 @@
 {
   "name": "clarify",
-  "version": "1.0.0-alpha.20",
+  "version": "1.0.0-alpha.21",
   "private": true,
   "type": "module",
   "scripts": {