diff --git a/docs/operations/auto-rollback.md b/docs/operations/auto-rollback.md index 4663e39f..0267cb93 100644 --- a/docs/operations/auto-rollback.md +++ b/docs/operations/auto-rollback.md @@ -1,10 +1,10 @@ # Automated rollback on CI failure -This runbook covers the automation for GitHub issue **#236**: detect a failed production deploy, revert to the last known-good version, notify Slack/Discord, and open an incident ticket. +This runbook covers the automation for GitHub issue *#236*: detect a failed production deploy, revert to the last known-good version, notify Slack/Discord, and open an incident ticket. ## What triggers a rollback -The workflow [`.github/workflows/auto-rollback.yml`](../../.github/workflows/auto-rollback.yml) runs when: +The workflow [.github/workflows/auto-rollback.yml] (../../.github/workflows/auto-rollback.yml) runs when: - The **Deploy - Frontend to Vercel and Artifacts** workflow finishes with `failure` or `timed_out` on `main`, or - An operator starts **Auto-rollback on deploy failure** from the Actions UI (defaults to dry-run). @@ -42,6 +42,32 @@ yarn ops:rollback --dry-run Or dispatch the workflow with **dry_run** enabled. +## Reading Vercel deployment failure logs + +When a production deploy fails, the rollback automation uses Vercel's deployment status. To investigate the root cause: + +1. **List recent deployments** to identify the failed deployment ID: + ```bash + vercel ls --prod + ``` + Or use the dashboard: https://vercel.com/dashboard → select project ↔ **Deployments**. + +2. **Inspect build logs** for the failed deployment. CLI: + ```bash + vercel logs + ``` + In the dashboard, open the deployment and choose **Build Logs**. Look for the first error (often TypeScript, Babel, or dependency errors). + +3. **Check runtime logs** if the deployment built but failed health checks: + ```bash + vercel logs --json + ``` + Search for `unhandled rejection`, `error`, `ECONNREFUSED`, or timeout messages. + +4. **Cross-reference environment variables** (required secrets above) to ensure the Vercel project has the same values as CI. + +Keep the logs with the incident ticket for later analysis. + ## After a rollback 1. Confirm `/api/health` and `/api/status` are green. diff --git a/docs/operations/runbook.md b/docs/operations/runbook.md index 267fc09e..ba5df246 100644 Binary files a/docs/operations/runbook.md and b/docs/operations/runbook.md differ diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 877e6725..db5ab94a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -276,3 +276,23 @@ console.log(window.stellar); // Verify Web Crypto API support console.log(window.crypto && window.crypto.subtle ? "WebCrypto OK" : "WebCrypto Unavailable"); ``` + +### 8.4 Failed Vercel Deployment Rollback Runbook + +Use this runbook when a production deployment fails or auto-rolls back in Vercel. + +#### Reading Deployment Failure Logs + +1. Open the Vercel project dashboard and go to **Deployments**. +2. Select the failed deployment (marked with a red **Error** badge). +3. Click **Inspect Deployment** and open the **Build Logs** tab. +4. Search for `Error:`, `Failed to compile`, or `Command exited with code` to find the root cause. +5. For runtime failures after the build succeeds, open the **Runtime Logs** tab and filter by the failing deployment. + +#### Rolling Back to the Last Known Good Deployment + +1. In the **Deployments** tab, identify the most recent deployment with a green **Ready** badge. +2. Click the overflow menu (three dots) on that deployment and choose **Promote to Production** (or **Redeploy** for the same production URL). +3. Confirm the promotion in the dialog and wait for the promotion to complete. +4. Verify the production URL returns HTTP 200 and the application loads in the browser. +5. If the rollback itself fails, create a GitHub issue tagged `runbook:vercel-rollback` with the failed deployment URL, build log excerpt, and target deployment hash. diff --git a/runbook.md b/runbook.md index d6d8cda0..8dcbefc7 100644 --- a/runbook.md +++ b/runbook.md @@ -271,3 +271,29 @@ We apologize for the disruption and appreciate your patience while we work to fu Regards, [promptmint/IT Team] + +--- + +# Failed Vercel Deployment Rollback + +Use this runbook when a Vercel deployment fails and Vercel automatically rolls back to the last successful deployment, or when you need to manually roll back a bad deployment. + +## Rollback Procedure + +1. Go to the Vercel project dashboard. +2. Open the **Deployments** tab. +3. Locate the failed deployment and note the error status. +4. If Vercel did not automatically roll back, open the last successful deployment. +5. Select **Promote to Production** to redeploy the last known-good version. +6. Confirm the new production deployment reaches **Ready**. +7. Verify the application health endpoint, critical user flows, and environment variables after rollback. + +## Reading Deployment Failure Logs + +1. Open the failed deployment in Vercel. +2. Select **Logs** to view build/runtime logs. +3. Look for `ERROR`, `Failed`, `Rolling back`, or non-zero exit codes near the failure point. +4. For build failures, check dependency installation, environment variable resolution, and compile output. +5. For runtime failures, inspect function logs for startup crashes, missing secrets, or invalid configuration. +6. If the failure is intermittent, download or copy the logs immediately because Vercel retains deployment logs for a limited time. +7. Attach the relevant log lines and deployment URL to the incident record before closing the issue.