diff --git a/.github/DEPLOYMENT.md b/.github/DEPLOYMENT.md index 7c70c46..60b896f 100644 --- a/.github/DEPLOYMENT.md +++ b/.github/DEPLOYMENT.md @@ -1,11 +1,24 @@ # Deployment Configuration — GitHub Secrets -Secrets consumed by `.github/workflows/deploy-staging.yml` and -`.github/workflows/e2e-staging-tests.yml`. Every entry below was cross-checked -against those workflow files; nothing here is aspirational. +Secrets consumed by the workflows in `.github/workflows/`. Every entry below was +cross-checked against those files; nothing here is aspirational. Configure at: `Settings` → `Secrets and variables` → `Actions`. +> ### ⚠️ Application secrets no longer come from here +> +> Staging moved to the container deploy on 2026-08-12 (#484). The live workflow, +> `deploy-staging-containers.yml`, ships the compose files and runs +> `docker compose up` — **it writes no env file**. Every application secret +> (`MONGODB_URI`, `DATABASE_NAME`, `BETTER_AUTH_SECRET`, `OPENAI_API_KEY`, AWS, +> Cloudinary, …) is read from `/opt/auto-author/.env` **on the server**, which is +> maintained by hand and which no workflow touches. +> +> The GitHub secrets for those values now feed only `deploy-staging.yml.disabled` +> — the retired PM2 deploy, kept as a rollback path. **Changing them does not +> change what staging is running.** See `docs/STAGING-DEPLOYMENT.md` for the box +> setup and `docs/DATABASE_CONNECTION_STANDARD.md` for the DB values. + > **Scope matters.** The staging test credentials (`TEST_USER_EMAIL`, > `TEST_USER_PASSWORD`) are **environment** secrets on the `staging` > environment, not repository secrets — a workflow that reads them must declare @@ -13,15 +26,36 @@ Configure at: `Settings` → `Secrets and variables` → `Actions`. --- +## Which workflow uses what + +| Workflow | State | Secrets it reads | +|---|---|---| +| `deploy-staging-containers.yml` | **live** | `TS_CLIENT_ID`, `TS_AUTH_SECRET`, `SSH_KEY`, `USER`, `STAGING_TS_HOST` (falls back to `HOST`) | +| `e2e-staging-tests.yml` | **live** | `TEST_USER_EMAIL`, `TEST_USER_PASSWORD` (staging environment), `SLACK_WEBHOOK_URL`, `DISCORD_WEBHOOK_URL` | +| `build-images.yml` | **live** | `GITHUB_TOKEN` (automatic) | +| `glm-review.yml` | **live** | `ZHIPU_API_KEY` | +| `deploy-staging.yml.disabled` | retired (rollback) | everything in the "Required/Recommended/Optional" sections below | +| `deploy-production.yml.disabled` | retired | — | + +**Server access** below is live — the container deploy reads those secrets from +GitHub on every run. Everything after it documents **application** values: for +the live deploy those belong in the box's `.env`, and the GitHub secret of the +same name feeds only the disabled PM2 path. + +--- + ## Required Secrets -### SSH & Server Access +### Server access | Secret | Description | Example | |--------|-------------|---------| | `SSH_KEY` | Private SSH key for server access | `-----BEGIN OPENSSH PRIVATE KEY-----…` | -| `HOST` | Server hostname or IP address | `195.35.14.177` | | `USER` | SSH username | `root` | +| `STAGING_TS_HOST` | Tailscale hostname for the box; preferred over `HOST` | `staging-vps` | +| `HOST` | Public hostname or IP — fallback when `STAGING_TS_HOST` is unset | `195.35.14.177` | +| `TS_CLIENT_ID` | Tailscale OAuth client id — the deploy runs over Tailscale (#485, #489) | — | +| `TS_AUTH_SECRET` | Tailscale OAuth client secret; needs the client's full tag set (#490) | — | ### Application URLs @@ -40,9 +74,19 @@ Configure at: `Settings` → `Secrets and variables` → `Actions`. | Secret | Description | Example | |--------|-------------|---------| -| `MONGODB_URI` | MongoDB connection string — written to `.env` as `DATABASE_URL` | `mongodb+srv://user:pass@cluster.mongodb.net` | +| `MONGODB_URI` | Connection string, **no database name in the path** | `mongodb+srv://user:pass@cluster.mongodb.net/` | | `DATABASE_NAME` | Database name | `auto_author_staging` | +For the live container deploy these are keys in `/opt/auto-author/.env`, read by +`docker-compose.yml` (which also remaps `MONGODB_URI` to the frontend's +`DATABASE_URL`). The PM2 path wrote the same value into `.env` under the key +`DATABASE_URL` instead — worth knowing when reading an older box. + +A database name in the URI path breaks Atlas auth (it becomes `defaultauthdb`, +which `authSource` then defaults to) and is ignored by the app regardless. +Percent-encode the password. Full rules and the rotation runbook: +`docs/DATABASE_CONNECTION_STANDARD.md`. + Staging uses MongoDB Atlas. When the server's IP changes, add the new address to the Atlas allowlist or every deploy health-check will fail. @@ -156,7 +200,12 @@ regardless. --- -## Environment Files the Workflow Creates +## Environment Files — the retired PM2 layout + +> These files are what `deploy-staging.yml.disabled` **used to** regenerate from +> secrets on every deploy. The container deploy creates nothing: it reads the +> single `/opt/auto-author/.env` listed in `docker-compose.yml`. Kept here +> because the rollback path still produces this layout. ### Backend: `/opt/auto-author/current/backend/.env` diff --git a/.github/workflows/deploy-staging-containers.yml b/.github/workflows/deploy-staging-containers.yml index 92f8565..9ecde48 100644 --- a/.github/workflows/deploy-staging-containers.yml +++ b/.github/workflows/deploy-staging-containers.yml @@ -1,16 +1,10 @@ name: Deploy Staging (Containers) -# Container deploy for staging (#427, items 5-6). DISABLED until the first -# cutover is done with a human present — see the checklist in -# docs/STAGING-DEPLOYMENT.md. Enable by renaming to -# deploy-staging-containers.yml and disabling the PM2 workflow -# (deploy-staging.yml -> .disabled), not before. +# Container deploy for staging (#427, items 5-6). LIVE since 2026-08-12 (#484), +# which retired the PM2 workflow to deploy-staging.yml.disabled. Box setup and +# the rollback procedure: docs/STAGING-DEPLOYMENT.md. # -# The .disabled suffix follows this repo's existing convention -# (deploy-production.yml.disabled). GitHub only loads .yml/.yaml, so this file -# is inert as-is. -# -# What changes vs the PM2 deploy it replaces: +# What changes vs the PM2 deploy it replaced: # - no npm ci / uv sync on the VPS; images are built in CI (build-images.yml) # and pulled by tag # - no releases/ symlink dance; the image tag IS the release identifier diff --git a/CLAUDE.md b/CLAUDE.md index 85b16ed..a787a55 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,8 +88,8 @@ Files under 500 lines. Never hardcode secrets. Tests before implementation. WCAG - Staging: https://dev.autoauthor.app (frontend), https://api.dev.autoauthor.app (backend). SSH as `root`, keys are local. - **Shared box — other apps run here.** Backend is on 8000, frontend on 3002, but check nginx for current truth before assuming a port is free. -- PM2-managed with a symlinked `current` release dir; when things look out of sync, check that the symlink points where you think it does. -- Deployment scripts live in the git workflow directories. +- **Containers since 2026-08-12 (#484)** — `deploy-staging-containers.yml` ships the compose files and pulls a `sha-` image tag; the tag *is* the release id. PM2 is retired (`deploy-staging.yml.disabled`, kept for rollback). +- **Application secrets live in `/opt/auto-author/.env` on the box, not in GitHub secrets.** No workflow writes that file. Changing the `MONGODB_URI` repo secret changes nothing that runs. See `docs/STAGING-DEPLOYMENT.md`. --- diff --git a/docs/DATABASE_CONNECTION_STANDARD.md b/docs/DATABASE_CONNECTION_STANDARD.md index ef75f5d..298b380 100644 --- a/docs/DATABASE_CONNECTION_STANDARD.md +++ b/docs/DATABASE_CONNECTION_STANDARD.md @@ -1,7 +1,7 @@ # Database Connection Standard -**Last Updated**: 2025-12-29 -**Status**: ✅ FINAL - All code is consistent with this standard +**Last Updated**: 2026-08-13 +**Status**: ✅ Current — enforced by code; the plumbing below matches the container deploy ## 🎯 The Standard (Non-Negotiable) @@ -16,192 +16,162 @@ DATABASE_NAME=auto_author_staging ``` +### Why this is not merely style + +A database name in the path becomes the driver's `defaultauthdb`, and +`authSource` **defaults to that** when it isn't given explicitly. Atlas users +live in `admin`, so `…mongodb.net/auto_author_staging` without an explicit +`?authSource=admin` authenticates against the wrong database and fails — which +looks exactly like a bad password, and is especially confusing during a +credential rotation. + +The name in the path is also *ignored* by both services regardless, because each +selects its database explicitly (see "How it works"). So it can only do harm. + ## 📋 Variable Naming Convention -| Context | Variable Name | Contains | Example | -|---------|---------------|----------|---------| -| **GitHub Secret** | `MONGODB_URI` | Connection URI (no DB) | `mongodb+srv://user:pass@cluster.mongodb.net/` | -| **GitHub Secret** | `DATABASE_NAME` | Database name only | `auto_author_staging` | -| **Environment Variable** | `DATABASE_URL` | Set from `MONGODB_URI` | `mongodb+srv://user:pass@cluster.mongodb.net/` | -| **Environment Variable** | `DATABASE_NAME` | Database name only | `auto_author_staging` | -| **Python Code** | `settings.DATABASE_URL` | Read from env var | `mongodb+srv://user:pass@cluster.mongodb.net/` | -| **Python Code** | `settings.DATABASE_NAME` | Read from env var | `auto_author_staging` | +| Context | Variable | Contains | +|---|---|---| +| `/opt/auto-author/.env` on the box | `MONGODB_URI` | Connection URI, **no** DB name | +| `/opt/auto-author/.env` on the box | `DATABASE_NAME` | Database name only | +| Backend container env | `MONGODB_URI`, `DATABASE_NAME` | Passed through by compose | +| Frontend container env | `DATABASE_URL`, `DATABASE_NAME` | `DATABASE_URL` is `MONGODB_URI` remapped | +| Python | `settings.MONGODB_URI` / `settings.DATABASE_NAME` | Read from env | -## 🔄 How It Works +`DATABASE_URL` is the backend's **legacy fallback**, used only when `MONGODB_URI` +is empty (`backend/app/core/config.py:53`). New configuration should set +`MONGODB_URI`. -### 1. GitHub Secrets (Repository Settings) -```yaml -# In GitHub repository secrets: -MONGODB_URI: mongodb+srv://frankbria:password@cluster.mongodb.net/ -DATABASE_NAME: auto_author_staging -``` +## 🔄 How it works + +### 1. The box `.env` is the source of truth + +Staging runs the container deploy (`.github/workflows/deploy-staging-containers.yml`), +which ships the compose files and runs `docker compose up`. **It writes no env +file.** The values come from `/opt/auto-author/.env`, maintained by hand on the +server. + +There is a `MONGODB_URI` GitHub secret, but it is referenced only by +`deploy-staging.yml.disabled` — the retired PM2 deploy. Changing it does not +affect what staging connects with. + +### 2. Compose passes it to both services -### 2. GitHub Actions Workflow ```yaml -# Workflows map secrets to environment variables: -env: - DATABASE_URL: ${{ secrets.MONGODB_URI }} # Note: Maps MONGODB_URI → DATABASE_URL - DATABASE_NAME: ${{ secrets.DATABASE_NAME }} +# docker-compose.yml +backend: + environment: + MONGODB_URI: ${MONGODB_URI:?MONGODB_URI is required} + DATABASE_NAME: ${DATABASE_NAME:?DATABASE_NAME is required} +frontend: + environment: + DATABASE_URL: ${MONGODB_URI:?MONGODB_URI is required} # better-auth's name for it + DATABASE_NAME: ${DATABASE_NAME:?DATABASE_NAME is required} ``` -### 3. Backend Configuration (config.py) -```python -class Settings(BaseSettings): - DATABASE_URL: str = "mongodb://localhost:27017" # Read from env var - DATABASE_NAME: str = "auto_author_test" # Read from env var -``` +The `:?` form makes compose fail fast with a named error rather than starting a +container that dies at first query. -### 4. Database Connection (base.py) -```python -# Connect to MongoDB cluster (no database specified in URI) -_client = AsyncIOMotorClient(settings.DATABASE_URL, ...) +### 3. Each service selects the database explicitly -# Explicitly select database +```python +# backend/app/db/base.py:33 _db = _client[settings.DATABASE_NAME] ``` -## ✅ Why This Pattern? - -### Flexibility -- **Same URI** works for multiple databases (dev, staging, prod) -- **Easy switching**: Change `DATABASE_NAME` without touching `MONGODB_URI` -- **Testing**: Different test databases without new connection strings +```typescript +// frontend/src/lib/auth.ts:156 — better-auth's MongoDB adapter +db = client.db(dbName); // dbName = process.env.DATABASE_NAME +``` -### Security -- **Separate concerns**: Connection credentials vs. database selection -- **Easier rotation**: Rotate database user password without changing database names +Neither reads the database from the URI path. That is what makes the standard +enforceable rather than aspirational. -### Clarity -- **Explicit**: Clear which database you're using -- **No ambiguity**: No confusion about whether database is in URI or not +## ⚠️ Escaping the password -## 📍 Where This Standard Applies +The URI goes into a dotenv file that compose reads **twice** — once literally for +`env_file:`, once with `${...}` interpolation for the `environment:` block, and +the interpolated value wins. Characters that break one path or the other: -### ✅ Already Consistent +| Character | Breaks | +|---|---| +| `$` | compose interpolation (would need `$$`) | +| `#` | dotenv — starts a comment, truncating the URI | +| space | dotenv parsing | +| `| & \` | the `sed` substitution in the legacy PM2 path | -1. **Backend Code** - - `backend/app/core/config.py` - Settings definition - - `backend/app/db/base.py` - Database connection - - `backend/app/populate_db_test_data.py` - Test data script - - `backend/.env.example` - Environment template +**Percent-encode the password** and all of this goes away — `%XX` contains none +of these. `@` → `%40`, `/` → `%2F`, `%` → `%25`, `:` → `%3A`. -2. **GitHub Workflows** - - `.github/workflows/tests.yml` - E2E test workflow - - `.github/workflows/deploy-staging.yml` - Staging deployment +## 🔁 Rotating the password -3. **Documentation** - - `docs/GITHUB_SECRETS_SETUP.md` - Secrets configuration guide - - `docs/DATABASE_CONNECTION_STANDARD.md` - This document +1. Rotate the user in Atlas. **The old password dies immediately** — the running + containers start failing on their next reconnect, not at the next deploy. +2. Percent-encode the new password into the URI. +3. Edit `MONGODB_URI` in `/opt/auto-author/.env` on the box. +4. `cd /opt/auto-author && docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d` + — **recreate, not restart**; a restarted container keeps its old environment. +5. Verify: `curl -s 127.0.0.1:8000/api/v1/health`. Since #333 this does a real + Mongo ping, so bad credentials surface here as a 503 naming the component. -## 🚫 Common Mistakes to Avoid +Optionally update the `MONGODB_URI` GitHub secret to keep the disabled PM2 +workflow from holding a dead credential — but note that leaves a second copy of a +live credential with no consumer. -### ❌ DON'T: Put database name in URI -```python -# WRONG - Database name in URI -DATABASE_URL = "mongodb+srv://user:pass@cluster.mongodb.net/mydb" -_client = AsyncIOMotorClient(DATABASE_URL) -_db = _client[DATABASE_NAME] # This would try to access DATABASE_NAME, not "mydb" -``` +## 🚫 Common mistakes -### ✅ DO: Keep them separate +### ❌ DON'T: put the database name in the URI ```python -# CORRECT - URI without database -DATABASE_URL = "mongodb+srv://user:pass@cluster.mongodb.net/" -_client = AsyncIOMotorClient(DATABASE_URL) -_db = _client[DATABASE_NAME] # Explicitly selects the database +# WRONG — becomes defaultauthdb, breaks Atlas auth, and is ignored anyway +MONGODB_URI = "mongodb+srv://user:pass@cluster.mongodb.net/mydb" ``` -### ❌ DON'T: Hardcode database name +### ❌ DON'T: hardcode the database name ```python -# WRONG - Hardcoded -_db = _client["auto_author_staging"] +_db = _client["auto_author_staging"] # WRONG +_db = _client[settings.DATABASE_NAME] # correct ``` -### ✅ DO: Use the setting -```python -# CORRECT - From settings -_db = _client[settings.DATABASE_NAME] -``` +### ❌ DON'T: assume the GitHub secret is what's running +The live deploy reads the box's `.env`. Confirm there before debugging further. -## 🔍 How to Verify Compliance +## 🔍 Verifying compliance -### Check GitHub Secrets ```bash -gh secret list | grep -E "MONGODB_URI|DATABASE_NAME" -``` +# On the box — key names and shape, password masked +sed -E 's|(//[^:]+:)[^@]+@|\1****@|' /opt/auto-author/.env | grep -E 'MONGODB_URI|DATABASE_NAME' -Should show: -``` -DATABASE_NAME Updated 2025-12-XX -MONGODB_URI Updated 2025-12-XX -``` - -### Check Environment Variables -```bash -# In backend directory -grep -E "DATABASE_URL|DATABASE_NAME" .env .env.example -``` - -Should show separate values: -``` -DATABASE_URL=mongodb://localhost:27017 -DATABASE_NAME=auto_author_test -``` - -### Check Code Usage -```bash -# Search for improper usage -grep -r "mongodb.*/" backend/app --include="*.py" | grep -v "mongodb://localhost" | grep -v "mongodb+srv://" +# Code still selects the DB explicitly +grep -n "DATABASE_NAME" backend/app/db/base.py frontend/src/lib/auth.ts ``` -Should return minimal results (only in comments/docs). +`MONGODB_URI` should end at the host (a trailing `/` is fine); anything after it +is a bug per this document. ## 📝 Examples -### Local Development (.env) ```bash -DATABASE_URL=mongodb://localhost:27017 +# Local development +MONGODB_URI=mongodb://localhost:27017 DATABASE_NAME=auto_author_dev -``` -### Staging Environment (GitHub Secrets) -```bash -MONGODB_URI=mongodb+srv://staging_user:pass@cluster.mongodb.net/ +# Staging (in /opt/auto-author/.env on the box) +MONGODB_URI=mongodb+srv://staging_user:pa%24%24word@cluster.mongodb.net/ DATABASE_NAME=auto_author_staging -``` - -### Production Environment (GitHub Secrets) -```bash -MONGODB_URI=mongodb+srv://prod_user:pass@cluster.mongodb.net/ -DATABASE_NAME=auto_author_prod -``` -### Testing Environment (pytest) -```bash -DATABASE_URL=mongodb://localhost:27017 -DATABASE_NAME=auto_author_test +# Tests — note TEST_MONGO_URI is the exception: it DOES carry the db name in +# the path, and backend/tests/conftest.py parses it from there. +TEST_MONGO_URI=mongodb://localhost:27017/auto-author-test ``` -## 🔒 Security Notes - -1. **Never commit** `MONGODB_URI` values to git -2. **Use different credentials** for each environment -3. **Limit permissions** on database users (no admin access for app) -4. **Rotate credentials** periodically (quarterly recommended) -5. **Monitor access** via MongoDB Atlas logs - -## 🤝 Enforcement - -**This standard is enforced by**: -- Code reviews (check PR diffs for violations) -- Integration tests (verify connection works) -- Documentation (this file serves as source of truth) +## 🔒 Security notes -**If you find inconsistencies**: -1. File an issue -2. Reference this document -3. Propose a fix that aligns with this standard +1. **Never commit** connection URIs to git; they live only in the box `.env`. +2. Use different credentials per environment. +3. Limit the database user's permissions — the app needs no admin rights. +4. Rotate periodically, following the runbook above. --- -**Questions?** Refer to `docs/GITHUB_SECRETS_SETUP.md` for implementation details. +**Related**: `.github/DEPLOYMENT.md` (which secrets each workflow uses), +`docs/STAGING-DEPLOYMENT.md` (box setup for the container deploy). diff --git a/docs/GITHUB_SECRETS_SETUP.md b/docs/GITHUB_SECRETS_SETUP.md deleted file mode 100644 index 883881f..0000000 --- a/docs/GITHUB_SECRETS_SETUP.md +++ /dev/null @@ -1,147 +0,0 @@ -# GitHub Secrets Configuration for E2E Tests - -This document explains the GitHub Secrets used by the E2E test workflow. - -## ✅ Existing Secrets (Already Configured) - -The E2E tests workflow uses the following secrets that are **already configured** in your GitHub repository: - -### 1. `MONGODB_URI` (Already exists) -**Description**: MongoDB Atlas connection string **WITHOUT database name** -**Format**: `mongodb+srv://:@.mongodb.net/` -**Example**: `mongodb+srv://user:pass@cluster.mongodb.net/` -**Usage**: Used by both staging deployment and E2E tests - -**IMPORTANT**: The database name is NOT included in this URI. The database is specified separately via `DATABASE_NAME`. - -**Note**: This secret is already configured and used by the `deploy-staging.yml` workflow. - -### 2. `DATABASE_NAME` (Already exists) -**Description**: Database name only (not part of the connection URI) -**Format**: Just the database name (e.g., `auto_author_staging`, `auto_author_test`) -**Example**: `auto_author_staging` -**Usage**: Explicitly specifies which database to use within the MongoDB cluster - -**IMPORTANT**: This is kept separate from `MONGODB_URI` for flexibility. The same connection URI can be used with different databases by changing only this value. - -**Note**: This secret is already configured and used by the `deploy-staging.yml` workflow. - -### 3. `BETTER_AUTH_SECRET` (Already exists) -**Description**: Secret key for JWT token signing -**Format**: Random string (minimum 32 characters, recommended 64+) - -**How to generate**: -```bash -python -c 'import secrets; print(secrets.token_urlsafe(64))' -``` - -**Security Notes**: -- Use a different secret for CI than production -- Never commit this to source control -- Rotate periodically - -### 4. `OPENAI_API_KEY` (Already exists) -**Description**: OpenAI API key for AI features -**Format**: `sk-...` - -**Note**: This secret is already configured and used by the `deploy-staging.yml` workflow. - -## How These Secrets Are Used in E2E Tests - -The secrets are automatically used by the E2E test workflow without any additional configuration needed. - -## Workflow Configuration - -The E2E tests workflow (`.github/workflows/tests.yml`) uses these secrets as follows: - -```yaml -env: - DATABASE_URL: ${{ secrets.MONGODB_URI }} - DATABASE_NAME: ${{ secrets.DATABASE_NAME }} - BETTER_AUTH_SECRET: ${{ secrets.BETTER_AUTH_SECRET }} - BETTER_AUTH_URL: http://localhost:3000 - BYPASS_AUTH: true - E2E_ALLOW_BYPASS: '1' # required alongside BYPASS_AUTH since #272 (frontend) and #307 (backend) - OPENAI_AUTOAUTHOR_API_KEY: ${{ secrets.OPENAI_API_KEY }} -``` - -**Note**: These are the same secrets used by the staging deployment workflow, ensuring consistency across environments. - -## MongoDB Atlas SSL/TLS Configuration - -The backend automatically uses SSL/TLS for MongoDB Atlas connections with the following configuration: - -- **Certificate Validation**: Uses `certifi` package for trusted CA certificates -- **TLS Enabled**: Automatically enabled for `mongodb+srv://` URIs -- **Timeouts**: - - Server selection: 30 seconds - - Connection: 20 seconds - - Socket operations: 20 seconds - -See `backend/app/db/base.py` for implementation details. - -## Testing the Configuration - -Since all secrets are already configured, you can trigger a workflow run immediately: - -1. **Trigger via push** (recommended): - ```bash - git add . - git commit -m "feat: Configure E2E tests with MongoDB Atlas SSL/TLS" - git push - ``` - -2. **Manual trigger** (alternative): - ```bash - gh workflow run tests.yml - ``` - -3. **Check workflow logs**: - - Go to **Actions** tab in GitHub - - Click on the workflow run - - Check "E2E Tests" job logs - - Look for "Backend is ready!" message within 60 seconds - -## Troubleshooting - -### Backend fails to start -**Symptom**: "Backend failed to start within 60 seconds" - -**Possible causes**: -1. Invalid MongoDB Atlas URL -2. Network connectivity issues -3. Invalid credentials -4. Database user permissions insufficient - -**Solution**: Check workflow logs for detailed error messages - -### SSL/TLS handshake errors -**Symptom**: SSL certificate verification failed - -**Solution**: Ensure you're using Python 3.13 and the `certifi` package is installed (already configured in `backend/pyproject.toml`) - -### Authentication errors -**Symptom**: 401 Unauthorized responses - -**Possible causes**: -1. `BETTER_AUTH_SECRET` not set or invalid -2. `BYPASS_AUTH` not properly configured - -**Solution**: Verify `BYPASS_AUTH=true` is set in workflow environment - -## Security Best Practices - -1. **Principle of Least Privilege**: Grant minimal necessary permissions -2. **Separate Environments**: Use different credentials for dev/staging/prod/CI -3. **Credential Rotation**: Rotate secrets periodically (quarterly recommended) -4. **Audit Logs**: Monitor MongoDB Atlas and OpenAI usage -5. **Cost Controls**: Set usage limits on OpenAI API key -6. **Network Security**: Use MongoDB Atlas IP allowlists if possible -7. **Secrets Scanning**: Enable GitHub secret scanning (automatically enabled for public repos) - -## Related Documentation - -- [GitHub Actions Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) -- [MongoDB Atlas Security](https://www.mongodb.com/docs/atlas/security/) -- [Better Auth Documentation](https://www.better-auth.com/) -- [OpenAI API Best Practices](https://platform.openai.com/docs/guides/safety-best-practices) diff --git a/docs/STAGING-DEPLOYMENT.md b/docs/STAGING-DEPLOYMENT.md index 6cb4bb0..979c763 100644 --- a/docs/STAGING-DEPLOYMENT.md +++ b/docs/STAGING-DEPLOYMENT.md @@ -1,791 +1,197 @@ -# Staging Server Deployment Guide +# Staging Deployment Guide -> **Status note (#427):** the container deploy is being introduced alongside the -> PM2 path documented below. Images are already built and published on every -> push to `main` (`.github/workflows/build-images.yml`), and the deploy workflow -> is written but **disabled** pending a first supervised cutover. See -> [Container deploy (#427)](#container-deploy-427) at the end of this document. -> -> Until that cutover happens the PM2 instructions below are still the live path -> and remain correct. Do not follow both. +**Last Updated**: 2026-08-13 +**Deploy path**: containers, live since 2026-08-12 (#484) -**Last Updated**: 2025-10-19 -**Target Server**: frankbria-inspiron-7586 -**Purpose**: Stable sprint demo and integration testing environment +Staging is https://dev.autoauthor.app (frontend) and https://api.dev.autoauthor.app +(backend). It runs as Docker containers on a **shared VPS** — other applications +live on the same box, so every step here is a check, not an assumption. ---- - -## 📋 Table of Contents - -1. [Overview](#overview) -2. [Prerequisites](#prerequisites) -3. [Initial Server Setup](#initial-server-setup) -4. [Automated Deployment](#automated-deployment) -5. [Manual Deployment](#manual-deployment) -6. [Service Management](#service-management) -7. [Troubleshooting](#troubleshooting) -8. [CI/CD Integration](#cicd-integration) +> The PM2/rsync deploy this replaced (build-on-the-VPS, `releases/` symlinked +> to `current`) is retired. Its workflow is kept as `deploy-staging.yml.disabled` +> for rollback; see [Falling back to PM2](#falling-back-to-pm2). --- -## Overview - -The staging environment serves as: -- **Sprint Demo Environment**: Stable builds for stakeholder demonstrations -- **Integration Testing**: Full-stack testing before production -- **QA Environment**: Manual and automated testing -- **Pre-production Validation**: Final checks before production deployment - -### Architecture +## Architecture ``` -frankbria-inspiron-7586 (Staging Server) -├── Frontend (Next.js) → Port 3002 -├── Backend (FastAPI) → Port 8000 -└── PostgreSQL Database → Port 5432 +Shared VPS +├── nginx (:80/:443, TLS termination + security headers; CORS itself comes +│ from the backend's CORSMiddleware / BACKEND_CORS_ORIGINS) +│ ├── dev.autoauthor.app → 127.0.0.1:3002 (frontend container) +│ └── api.dev.autoauthor.app → 127.0.0.1:8000 (backend container) +├── /opt/auto-author/ +│ ├── docker-compose.yml shipped by the deploy +│ ├── docker-compose.staging.yml shipped by the deploy +│ └── .env NOT shipped — hand-maintained secrets +└── MongoDB: Atlas (external, IP-allowlisted) ``` -### Network Exposure (#189) - -The staging VPS is shared with other applications. Only nginx may face the -internet; the app processes must not be directly reachable: - -- **Loopback binding (primary)**: both PM2 apps bind `127.0.0.1` — - uvicorn via `--host 127.0.0.1` and Next via `next start -H 127.0.0.1` - (see `ecosystem.config.template.js`). nginx on the same box proxies - `dev.autoauthor.app` → `127.0.0.1:3002` and `api.dev.autoauthor.app` → - `127.0.0.1:8000`, terminating TLS and applying CORS/security headers. -- **Host firewall (defense-in-depth)**: ufw is active with default-deny - inbound; only 22/80/443 are allowed. Ports 8000/3002 are dropped at the - host even if a process ever binds `0.0.0.0` again. - Verified 2026-07-10: `curl http://:8000` and `:3002` time out - from off-box while `https://dev.autoauthor.app` serves normally. -- **Verify after deploy**: `ss -tlnp | grep -E ':8000|:3002'` must show only - `127.0.0.1` binds; health checks use `http://localhost:...` and are - unaffected. - ---- +Images are built in CI by `build-images.yml` and published to GHCR as +`sha-` and `staging`. The **image tag is the release identifier** — there +is no release directory. -## Prerequisites +### Network exposure (#189) -### Local Machine +The box is shared, so only nginx may face the internet: -- [x] SSH access to staging server -- [x] Git repository cloned -- [x] Node.js 18+ installed -- [x] Python 3.11+ installed -- [x] uv package manager installed - -### Staging Server Requirements - -**Software:** -- Ubuntu 20.04 LTS or newer -- Node.js 18+ (for frontend) -- Python 3.11+ (for backend) -- PostgreSQL 14+ (for database) -- PM2 (optional, for process management) -- Nginx (optional, for reverse proxy) - -**Resources:** -- 4GB RAM minimum (8GB recommended) -- 20GB disk space minimum -- Network: Ports 3002, 8000 accessible +- **Loopback binding (primary)** — both containers publish to loopback only: + `127.0.0.1:8000:8000` and `127.0.0.1:3002:3002` in `docker-compose.yml`. A bare + `8000:8000` would expose the app on the public interface past nginx's TLS and + headers; don't. +- **Host firewall (defense in depth)** — ufw is active, default-deny inbound, + only 22/80/443 allowed. Verified 2026-07-10: off-box `curl` to `:8000` and + `:3002` times out while `https://dev.autoauthor.app` serves normally. +- **Verify after a deploy** — `ss -tlnp | grep -E ':8000|:3002'` must show only + `127.0.0.1` binds. --- -## Initial Server Setup - -### Step 1: SSH Key Setup - -```bash -# On local machine -ssh-keygen -t ed25519 -C "staging-deployment" - -# Copy to staging server -ssh-copy-id frankbria@frankbria-inspiron-7586 - -# Test connection -ssh frankbria@frankbria-inspiron-7586 echo "SSH connection successful" -``` - -### Step 2: Install Dependencies on Staging Server - -```bash -# SSH into staging server -ssh frankbria@frankbria-inspiron-7586 - -# Update system -sudo apt update && sudo apt upgrade -y +## The box `.env` is the source of truth -# Install Node.js 18 LTS -curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - -sudo apt install -y nodejs - -# Install Python 3.11 -sudo apt install -y python3.11 python3.11-venv python3-pip - -# Install uv (Python package manager) -pip3 install uv - -# Install PM2 (process manager) -sudo npm install -g pm2 - -# Install PostgreSQL 14 -sudo apt install -y postgresql postgresql-contrib - -# (Optional) Install Nginx for reverse proxy -sudo apt install -y nginx -``` +`/opt/auto-author/.env` carries every application secret. **No workflow writes +it.** The deploy ships compose files and runs `docker compose up`; the values +come from that file. -### Step 3: Create Deployment Directory +Compose asserts the required ones with `${VAR:?...}`, so a missing key fails the +deploy with a named error instead of starting a container that dies at first +query: -```bash -# On staging server -mkdir -p /home/frankbria/staging/auto-author -cd /home/frankbria/staging/auto-author ``` - -### Step 4: Configure PostgreSQL - -```bash -# On staging server -sudo -u postgres psql - -# In PostgreSQL console: -CREATE DATABASE autoauthor_staging; -CREATE USER autoauthor_staging WITH PASSWORD 'your_secure_password'; -GRANT ALL PRIVILEGES ON DATABASE autoauthor_staging TO autoauthor_staging; -\q +MONGODB_URI, DATABASE_NAME, BETTER_AUTH_SECRET, OPENAI_API_KEY ``` -### Step 5: Environment Variables +Everything else in the file is passed through wholesale via `env_file:` — the app +also needs `AWS_*`, `CLOUDINARY_*`, `BETTER_AUTH_ISSUER` and +`BACKEND_CORS_ORIGINS`, which an explicit allowlist would have silently dropped. -Create `.env` files on staging server: - -**Backend** (`/home/frankbria/staging/auto-author/current/backend/.env`): +Editing a value requires **recreating** the containers, not restarting them: ```bash -# Database -DATABASE_URL=postgresql://autoauthor_staging:your_password@localhost:5432/autoauthor_staging - -# Authentication (Clerk) -CLERK_JWT_PUBLIC_KEY=-----BEGIN PUBLIC KEY----- -... (copy from local .env) ... ------END PUBLIC KEY----- - -# OpenAI -OPENAI_API_KEY=sk-... (copy from local .env) - -# Environment -ENVIRONMENT=staging -DEBUG=false +cd /opt/auto-author +docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d ``` -**Frontend** (`/home/frankbria/staging/auto-author/current/frontend/.env.local`): - -```bash -# API Configuration -NEXT_PUBLIC_API_URL=http://frankbria-inspiron-7586:8000/api/v1 - -# Authentication (Clerk) -NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_test_... (copy from local) -CLERK_SECRET_KEY=sk_test_... (copy from local) - -# Environment -NEXT_PUBLIC_ENVIRONMENT=staging -``` +Database values have their own rules (no db name in the URI, percent-encode the +password, rotation runbook): `docs/DATABASE_CONNECTION_STANDARD.md`. --- -## Automated Deployment - -> **Note**: Deployment is now fully automated via GitHub Actions. Legacy shell scripts have been removed. - -### Deployment via GitHub Actions - -Deployments to staging are handled automatically by the **Deploy to Staging** workflow (`.github/workflows/deploy-staging.yml`). - -**Automatic Triggers:** -- Push to `develop` branch -- After successful test suite completion on `develop` - -**Manual Trigger:** -```bash -# Via GitHub CLI -gh workflow run deploy-staging.yml +## One-time setup on the box -# Or use GitHub UI: -# Actions tab → Deploy to Staging → Run workflow -``` - -**The workflow automatically:** -1. ✅ Runs pre-flight checks -2. ✅ Executes test suites (frontend + backend) -3. ✅ Builds production frontend -4. ✅ Creates deployment package -5. ✅ Transfers to staging server via SSH -6. ✅ Extracts and sets up environment -7. ✅ Installs dependencies -8. ✅ Restarts services with PM2 -9. ✅ Runs health checks -10. ✅ Executes smoke tests -11. ✅ Sends Slack notification (if configured) - -### Required GitHub Secrets - -Ensure the following secrets are configured in repository settings: - -``` -SSH_KEY # Private SSH key for staging server -HOST # Staging server hostname -USER # SSH user -API_URL # Backend API URL -FRONTEND_URL # Frontend URL -CLERK_PUBLISHABLE_KEY -CLERK_SECRET_KEY -SLACK_WEBHOOK_URL # (Optional) for notifications -``` - -### Monitoring Deployments - -```bash -# View deployment status -gh run list --workflow=deploy-staging.yml - -# View logs for latest deployment -gh run view --log - -# Watch deployment in real-time -gh run watch -``` +1. **Ports.** Confirm 8000 and 3002 are free, or held only by this app: + `sudo ss -ltnp | grep -E ':(8000|3002)'` +2. **Docker.** Daemon installed and running; note containers belonging to other + applications: `docker ps -a` +3. **Directory.** `mkdir -p /opt/auto-author`, then place the two compose files + and the `.env`. Secrets live only in that file — never baked into an image. +4. **Registry access.** If the GHCR packages are private, `docker login ghcr.io` + with a token carrying `read:packages`. +5. **nginx.** Upstreams point at `127.0.0.1:8000` / `127.0.0.1:3002`. Unchanged + from the PM2 setup — which is why those ports were kept. --- -## Manual Deployment - -If automated deployment fails or you need manual control: - -### Step 1: Prepare Package Locally - -```bash -# Run tests -cd frontend && npm test -cd ../backend && uv run pytest - -# Build frontend -cd ../frontend && npm run build - -# Create tar archive -cd .. -tar -czf auto-author.tar.gz \ - --exclude='node_modules' \ - --exclude='.venv' \ - --exclude='.next' \ - --exclude='*.log' \ - frontend/ backend/ docs/ scripts/ -``` - -### Step 2: Transfer to Staging +## Deploying -```bash -scp auto-author.tar.gz frankbria@frankbria-inspiron-7586:/home/frankbria/staging/ -``` +Trigger **Deploy Staging (Containers)** via workflow dispatch with an explicit +`image_tag` (e.g. `sha-3931169`) published by `build-images.yml`. The workflow: -### Step 3: Extract on Staging +1. connects over Tailscale (#485, #489) and ships the two compose files by `scp`, + retrying with a cool-down — first SSH contact on this box intermittently gets + dropped during host-key discovery (#162); +2. `docker compose … pull` then `up -d --remove-orphans` — `pull` always goes to + the registry, so re-running the same tag still picks up a rebuilt image + instead of reusing a stale local layer; +3. polls `/api/v1/health` and `/` for up to 150s, dumping the last 60 log lines + and failing the job if either never returns 200; +4. prunes images older than 14 days — deliberately **after** the health check, so + a rollback target is never deleted first. -```bash -ssh frankbria@frankbria-inspiron-7586 +`concurrency: deploy-staging` with `cancel-in-progress: false` means two deploys +can never race on the box. -cd /home/frankbria/staging/auto-author - -# Backup current -mv current "backup-$(date +%Y%m%d-%H%M%S)" - -# Extract new -mkdir current -cd current -tar -xzf ../../auto-author.tar.gz -``` - -### Step 4: Setup Dependencies - -```bash -# Frontend -cd frontend -npm install --production - -# Backend -cd ../backend -uv venv -source .venv/bin/activate -uv pip install -r requirements.txt -``` - -### Step 5: Run Database Migrations +### Verifying ```bash -cd backend -source .venv/bin/activate -uv run alembic upgrade head +curl -s 127.0.0.1:8000/api/v1/health # real dependency checks since #333 +curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:3002/ +curl -s -o /dev/null -w '%{http_code}\n' https://dev.autoauthor.app/ ``` -### Step 6: Start Services +`/health` pings Mongo and asserts required secrets are present, so a +misconfigured release returns 503 naming the failing component rather than a +misleading 200. The staging E2E suite (`npm run test:e2e:staging`) uses real auth +and is the check that actually exercises the deployed stack. -**Option A: With PM2 (Recommended)** +### Rolling back -```bash -# Backend -cd backend -pm2 start "uvicorn app.main:app --host 127.0.0.1 --port 8000" \ - --name "auto-author-backend-staging" - -# Frontend -cd ../frontend -pm2 start "npm start -- -H 127.0.0.1" --name "auto-author-frontend-staging" - -# Save PM2 configuration -pm2 save -pm2 startup # Follow instructions to enable on boot -``` - -**Option B: Manual (for testing)** - -```bash -# Backend -cd backend -source .venv/bin/activate -nohup uvicorn app.main:app --host 127.0.0.1 --port 8000 > backend.log 2>&1 & - -# Frontend -cd ../frontend -nohup npm start -- -H 127.0.0.1 > frontend.log 2>&1 & -``` - ---- - -## Service Management - -### PM2 Commands - -```bash -# View all services -pm2 list - -# View logs -pm2 logs auto-author-backend-staging -pm2 logs auto-author-frontend-staging - -# Restart services -pm2 restart auto-author-backend-staging -pm2 restart auto-author-frontend-staging -pm2 restart all - -# Stop services -pm2 stop all - -# Delete services -pm2 delete all -``` - -### Manual Process Management - -```bash -# Find processes -ps aux | grep uvicorn -ps aux | grep "npm start" - -# Kill processes (if started manually) -kill $(cat /home/frankbria/staging/auto-author/current/backend/backend.pid) -kill $(cat /home/frankbria/staging/auto-author/current/frontend/frontend.pid) -``` - -### Database Management +Re-run the deploy with an earlier tag. That is the whole procedure: ```bash -# Connect to database -psql -U autoauthor_staging -d autoauthor_staging - -# Backup database -pg_dump -U autoauthor_staging autoauthor_staging > backup-$(date +%Y%m%d).sql - -# Restore database -psql -U autoauthor_staging autoauthor_staging < backup-20251019.sql +cd /opt/auto-author +IMAGE_TAG=sha- docker compose \ + -f docker-compose.yml -f docker-compose.staging.yml up -d ``` --- ## Troubleshooting -### Services Won't Start - -**Backend:** - -```bash -# Check logs -cd /home/frankbria/staging/auto-author/current/backend -cat backend.log - -# Common issues: -# 1. Database connection failed -# - Verify DATABASE_URL in .env -# - Check PostgreSQL is running: sudo systemctl status postgresql -# -# 2. Port 8000 already in use -# - Find process: lsof -i :8000 -# - Kill it: kill $(lsof -t -i :8000) -# -# 3. Missing dependencies -# - Reinstall: uv pip install -r requirements.txt -``` - -**Frontend:** - -```bash -# Check logs -cd /home/frankbria/staging/auto-author/current/frontend -cat frontend.log - -# Common issues: -# 1. Port 3002 already in use -# - Find process: lsof -i :3002 -# - Kill it: kill $(lsof -t -i :3002) -# -# 2. Build artifacts missing -# - Rebuild: npm run build -# -# 3. Environment variables missing -# - Check .env.local exists -# - Verify NEXT_PUBLIC_API_URL points to backend -``` - -### Health Check Failures - -```bash -# Test backend health -curl http://localhost:8000/api/v1/health - -# Test frontend -curl http://localhost:3002 - -# Check database connection -psql -U autoauthor_staging -d autoauthor_staging -c "SELECT 1" -``` - -### Deployment Script Issues - -```bash -# SSH connection fails -ssh-keygen -f "$HOME/.ssh/known_hosts" -R "frankbria-inspiron-7586" -ssh-keygen -R frankbria-inspiron-7586 -ssh-copy-id frankbria@frankbria-inspiron-7586 - -# Permission denied -chmod +x scripts/deploy-staging.sh - -# Package transfer fails -# Check disk space on staging: -ssh frankbria@frankbria-inspiron-7586 'df -h' -``` - ---- - -## CI/CD Integration +**Compose exits with `MONGODB_URI is required`** — the box `.env` is missing that +key, or the deploy is running from a directory without it. This is the assertion +working; check `/opt/auto-author/.env`. -### GitHub Actions Workflow +**Backend health 503** — read the `checks` object in the response body; it names +the failing component. Mongo failures are usually a rotated password not yet +written to the box `.env`, or a VPS IP that left the Atlas allowlist. -Future CI/CD automation can use this workflow structure: +**Ports already in use** — `sudo ss -ltnp | grep -E ':(8000|3002)'`. Shared box: +confirm the holder is ours before killing anything. -```yaml -name: Deploy to Staging +**Container up but serving stale code** — the tag was reused. Deploy an explicit +`sha-` tag rather than `staging`. -on: - push: - branches: [ main, develop ] - workflow_dispatch: - -jobs: - deploy-staging: - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v3 - - - name: Setup Node.js - uses: actions/setup-node@v3 - with: - node-version: '18' - - - name: Setup Python - uses: actions/setup-python@v4 - with: - python-version: '3.11' - - - name: Install uv - run: pip install uv - - - name: Run Tests - run: | - cd frontend && npm test - cd ../backend && uv run pytest - - - name: Deploy to Staging - env: - STAGING_HOST: ${{ secrets.STAGING_HOST }} - STAGING_USER: ${{ secrets.STAGING_USER }} - SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }} - run: | - mkdir -p ~/.ssh - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519 - chmod 600 ~/.ssh/id_ed25519 - ./scripts/deploy-staging.sh -``` - -### Webhook Integration - -For automated deployments on git push: - -```bash -# On staging server, setup webhook listener -npm install -g webhook - -# Create webhook.json configuration -# Start webhook daemon to trigger deployments -``` - -### Monitoring Integration - -Future monitoring setup: - -- **Uptime Monitoring**: UptimeRobot, Pingdom -- **Error Tracking**: Sentry -- **Performance**: New Relic, DataDog -- **Logs**: ELK Stack, CloudWatch - ---- - -## Best Practices - -### Pre-Deployment Checklist - -- [ ] All tests passing locally -- [ ] Code reviewed and merged to main/develop -- [ ] Database migrations prepared (if any) -- [ ] Environment variables documented -- [ ] Backup current staging deployment -- [ ] Notify team of deployment window - -### Post-Deployment Verification - -- [ ] Health checks pass -- [ ] Smoke tests complete -- [ ] Database migrations applied -- [ ] Frontend loads correctly -- [ ] API endpoints respond -- [ ] Authentication works -- [ ] Critical user flows tested - -### Rollback Procedure +**Logs** ```bash -# On staging server -cd /home/frankbria/staging/auto-author - -# Stop current services -pm2 stop all - -# Restore previous deployment -mv current current-failed -mv backup-YYYYMMDD-HHMMSS current - -# Restart services -pm2 restart all - -# Verify -curl http://localhost:8000/api/v1/health -curl http://localhost:3002 +cd /opt/auto-author +docker compose -f docker-compose.yml -f docker-compose.staging.yml logs --tail=100 backend +docker compose -f docker-compose.yml -f docker-compose.staging.yml ps ``` --- -## Security Considerations +## Falling back to PM2 -### Secrets Management +The retired path still exists: `deploy-staging.yml.disabled` regenerates +`/opt/auto-author/current/backend/.env` and `frontend/.env.production` **from +GitHub secrets** on every run, and manages processes with PM2. To use it, stop +the containers first (`docker compose … down`), then re-enable the workflow. -- Never commit `.env` files to git -- Use different credentials for staging vs production -- Rotate credentials regularly -- Limit SSH access to authorized users +Two differences that bite: -### Network Security - -```bash -# (Optional) Configure UFW firewall -sudo ufw allow 22 # SSH -sudo ufw allow 3002 # Frontend -sudo ufw allow 8000 # Backend -sudo ufw enable -``` +- it writes the connection string under the key `DATABASE_URL`, not + `MONGODB_URI` — the backend accepts both, preferring `MONGODB_URI`; +- its secret values are whatever is in GitHub, which may have drifted from the + box `.env` that the containers have been using. -### SSL/TLS (Optional for internal staging) - -```bash -# Install Let's Encrypt -sudo apt install certbot python3-certbot-nginx - -# Generate certificate -sudo certbot --nginx -d staging.yourdomain.com -``` +Remove this path (and the `MONGODB_URI` / `DATABASE_NAME` GitHub secrets, and the +PM2 ecosystem template) once containers have run without a rollback long enough +to trust — #427 AC 8. --- ## Maintenance -### Regular Tasks - -**Weekly:** -- Review application logs for errors -- Check disk space usage -- Verify backups are working - -**Monthly:** -- Update system packages: `sudo apt update && sudo apt upgrade` -- Rotate logs: `pm2 flush` -- Review and clean old backups - -**As Needed:** -- Database optimization: `VACUUM ANALYZE` -- Clear old deployment backups -- Update Node.js/Python versions +**Weekly** — review container logs for errors, check disk space (images +accumulate), confirm Atlas backups. +**Monthly** — `apt update && apt upgrade` on the box; review the image prune +window. +**As needed** — re-check the Atlas IP allowlist after any VPS network change. --- -## Support - -### Logs Location - -```bash -# PM2 logs -~/.pm2/logs/ - -# Application logs -/home/frankbria/staging/auto-author/current/backend/backend.log -/home/frankbria/staging/auto-author/current/frontend/frontend.log - -# System logs -/var/log/postgresql/ -/var/log/nginx/ -``` - -### Contact - -For deployment issues: -- Check this documentation first -- Review deployment script output -- Check application logs -- Contact DevOps team - ---- - -**Maintained By**: Development Team -**Review Frequency**: Before each major release -**Last Review**: 2025-10-19 - - ---- - -## Container deploy (#427) - -The PM2/rsync path above builds on the VPS: `npm ci` and `uv sync` run on a box -shared with other applications, a `releases/` directory is symlinked to -`current`, and PM2 is stopped and restarted. That makes deploys depend on the -box's toolchain versions and disk, and makes rollback a matter of hoping an old -release directory's `node_modules` still match. - -The container path replaces that with immutable images. - -### What already exists - -| Piece | Where | State | -|---|---|---| -| Backend image | `backend/Dockerfile` | multi-stage, non-root `appuser`, uvicorn on 8000 | -| Frontend image | `frontend/Dockerfile` | multi-stage Next standalone, non-root `node`, 3002 | -| Compose base | `docker-compose.yml` | both services on 127.0.0.1:8000 / :3002 — the ports nginx already fronts | -| Build overlay | `docker-compose.build.yml` | build contexts, used by CI and locally | -| Staging overlay | `docker-compose.staging.yml` | pins both images to `${IMAGE_TAG}` | -| Image publishing | `.github/workflows/build-images.yml` | **live** — publishes `sha-` and `staging` on every push to `main` | -| Deploy workflow | `.github/workflows/deploy-staging-containers.yml.disabled` | **written, disabled** | - -`build:` deliberately lives in its own overlay rather than the base file, so a -deploy that cannot pull an image **errors** instead of quietly compiling on the -shared box — which is the behaviour this migration exists to remove. - -### One-time setup on the box, before the first deploy - -This is a **shared machine**. Every step is a check, not an assumption. - -1. **Ports.** Confirm 8000 and 3002 are free, or that only this app holds them: - `sudo ss -ltnp | grep -E ':(8000|3002)'` -2. **Docker.** Confirm the daemon is installed and running, and note any - containers already there that belong to other applications: - `docker ps -a` -3. **Directory.** `mkdir -p /opt/auto-author` and place `docker-compose.yml`, - `docker-compose.staging.yml`, and a `.env` carrying the variables listed in - `docker-compose.yml` (`MONGODB_URI`, `DATABASE_NAME`, `BETTER_AUTH_SECRET`, - `OPENAI_API_KEY`, …). Secrets live only in that file — never in an image. -4. **Registry access.** If the GHCR packages are private, - `docker login ghcr.io` with a token that has `read:packages`. -5. **nginx.** Confirm the upstreams still point at `127.0.0.1:8000` and - `127.0.0.1:3002`. They should need no change — that is why those ports were - kept. - -### First cutover - -Do this with a person watching, not from an automatic trigger. - -1. Pick a tag published by `build-images.yml`, e.g. `sha-3931169`. -2. Stop the PM2 processes for this app only: - `pm2 stop auto-author-backend auto-author-frontend` - (`pm2 list` first — other applications may be under the same PM2.) -3. Bring the containers up: - ```bash - cd /opt/auto-author - IMAGE_TAG=sha-3931169 docker compose \ - -f docker-compose.yml -f docker-compose.staging.yml up -d - ``` -4. Verify locally, then through nginx: - ```bash - curl -s http://127.0.0.1:8000/api/v1/health # {"status":"healthy",...} - curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3002/ - curl -s -o /dev/null -w '%{http_code}\n' https://dev.autoauthor.app/ - ``` -5. Run the staging E2E suite (`npm run test:e2e:staging`) — it uses real auth, - so it is the check that actually exercises the deployed stack. -6. If anything is wrong, roll back immediately: `docker compose … down` and - `pm2 start auto-author-backend auto-author-frontend`. The PM2 release - directory is untouched by any of the above. - -### Rollback - -Re-run the deploy with an earlier tag. That is the whole procedure: - -```bash -cd /opt/auto-author -IMAGE_TAG=sha- docker compose \ - -f docker-compose.yml -f docker-compose.staging.yml up -d -``` - -The image-prune step in the deploy workflow keeps 14 days of images and runs -only after a successful health check, so the previous tag is still present when -you need it. - -### Enabling the automated deploy - -Only after the manual cutover has been done and verified **twice**: - -1. Rename `deploy-staging-containers.yml.disabled` → - `deploy-staging-containers.yml` -2. Rename `deploy-staging.yml` → `deploy-staging.yml.disabled` -3. Trigger it via workflow dispatch with an explicit `image_tag` before wiring - any automatic trigger - -Only then remove the PM2 ecosystem template and the PM2 steps (#427 AC 8). +**Related**: `.github/DEPLOYMENT.md` (which workflow reads which secret), +`docs/DATABASE_CONNECTION_STANDARD.md` (DB values and rotation).