Skip to content

OPERATIONS.md

Latest commit

 

History

History
448 lines (313 loc) · 15.9 KB

OPERATIONS.md

File metadata and controls

448 lines (313 loc) · 15.9 KB
Raw

Operations & Testing

This guide covers running, verifying, and troubleshooting Nova ID.

Security and deployment requirements

  • API must only be reachable via Oathkeeper. Do not expose the API (port 8080) to the public internet. All client traffic must go through the gateway; the API trusts headers (X-User-ID, etc.) injected by Oathkeeper. See SECURITY_REPORT.md.
  • Hydra Admin API: The API calls Hydra Admin (login/consent accept) from the server. Set HYDRA_ADMIN_URL=http://hydra:4445 (or your internal Hydra Admin URL). Do not expose Hydra Admin (port 4445) to the public internet. In local dev, the rule internal-hydra-admin (if used) must only be reachable from the API service on the internal network.
  • Secrets: Generate all secrets (Kratos, Hydra) with e.g. openssl rand -base64 32. Never deploy with placeholder values. See .env.example.

Compose files

File Purpose
docker-compose.yml Base stack (Ory, API, frontends). No local-proxy.
docker-compose.local.yml Local override: adds local-proxy (port 80) for auth.ory.localhost, admin.ory.localhost, etc. Development only — never used in production.
docker-compose.production.yml Production override: Traefik labels, no exposed ports. Does not include local-proxy.
  • Dev (ports 5173/5174/5175): docker compose up -d
  • Dev with local domains + local-proxy: docker compose -f docker-compose.yml -f docker-compose.local.yml up -d or ./start-local.sh
  • Production: docker compose -f docker-compose.yml -f docker-compose.production.yml up -d or ./start-production.sh

Service management

Start / stop

# Start all services
docker compose up -d

# Stop all services
docker compose down

# Stop and remove volumes (full reset)
docker compose down -v

Wait 30–60 seconds after up for migrations and services to be ready.

Restart

# Restart all
docker compose restart

# Restart a single service
docker compose restart kratos
docker compose restart api
docker compose restart oathkeeper

Status and logs

# List running services
docker compose ps

# Logs (all services, follow)
docker compose logs -f

# Logs for one service
docker compose logs -f api
docker compose logs -f oathkeeper
docker compose logs -f kratos

# Last N lines
docker compose logs --tail 100 api

Health checks

Manual checks

Service Endpoint Expected
Kratos GET http://localhost:4434/health/ready {"status":"ok"}
Keto Read GET http://localhost:4466/health/ready {"status":"ok"}
Keto Write GET http://localhost:4467/health/ready {"status":"ok"}
Hydra GET http://localhost:4445/health/alive {"status":"ok"}
Oathkeeper GET http://localhost:4456/health/alive {"status":"ok"}
API GET http://localhost:8080/health {"status":"ok",...} 
curl -s http://localhost:4434/health/ready | jq .
curl -s http://localhost:4466/health/ready | jq .
curl -s http://localhost:4456/health/alive | jq .
curl -s http://localhost:8080/health | jq .

API via Oathkeeper

# Public (no auth)
curl -s http://localhost:4455/api/health | jq .
curl -s http://localhost:4455/api/public | jq .

# Protected (requires session cookie from browser login)
# Use browser dev tools to copy cookie, then:
curl -s -b "ory_kratos_session=..." http://localhost:4455/api/protected | jq .

Permissions

Bootstrapping the platform admin (OPL model)

Nova ID uses Ory Keto's OPL model. The namespaces are User, Platform, and App. Platform-level admin access is granted via the relation tuple Platform:nova#admins@user:<kratos-id>.

Automatic seed on boot

The keto-seed compose service runs automatically after keto and kratos are healthy on every docker compose up. It is idempotent and one-shot (restart: "no"). It reads all Kratos identities whose metadata_public.role == "platform_admin" and writes the corresponding Platform:nova#admins tuples to Keto.

Note: metadata_public.role is the authoritative field — the identity schema blocks traits.role via additionalProperties: false. Keto is the source of truth after seeding; metadata_public.role is only used for bootstrapping.

Manual seed run

# Run the seed script manually (joins the internal network automatically):
docker compose run --rm keto-seed

# Or with host-exposed ports (only when temporarily debugging with ports exposed):
KETO_WRITE_URL=http://localhost:4467 \
KRATOS_ADMIN_URL=http://localhost:4434 \
docker compose run --rm keto-seed

Grant platform_admin to a user

./scripts/assign-platform-admin-to-user.sh user@example.com

The script updates metadata_public.role in Kratos and writes the Platform:nova#admins tuple directly to Keto in one step — no separate keto-seed run is required. Re-running docker compose run --rm keto-seed is safe (idempotent) but only needed for reconciliation (e.g., after a Keto data reset).

Verify Keto OPL permission check

The correct check endpoint for OPL computed permissions in Keto v25.4.0 uses the legacy GET check endpoint — it fully resolves OPL permits:

# Check if a user has the computed permission 'administer' on Platform:nova
# (resolves the OPL permit, not just the direct relation)
docker run --rm --network nova-id-ory-internal curlimages/curl:latest -s \
  "http://keto:4466/relation-tuples/check?namespace=Platform&object=nova&relation=administer&subject_id=user:<KRATOS_IDENTITY_ID>"
# Response: {"allowed":true}  (admin)  or  {"allowed":false}  (non-admin)

# Check manage_users permission:
docker run --rm --network nova-id-ory-internal curlimages/curl:latest -s \
  "http://keto:4466/relation-tuples/check?namespace=Platform&object=nova&relation=manage_users&subject_id=user:<KRATOS_IDENTITY_ID>"

# List all Platform tuples:
docker run --rm --network nova-id-ory-internal curlimages/curl:latest -s \
  "http://keto:4466/relation-tuples?namespace=Platform" | jq .

Task 5 note (Oathkeeper): The verified endpoint for Oathkeeper's Keto authorizer is: GET http://keto:4466/relation-tuples/check?namespace=Platform&object=nova&relation=administer&subject_id=user:{subject} → {"allowed":true|false}

Verify Keto tuples

# List relation tuples (run from host — requires ports temporarily exposed, or use docker run form above)
docker run --rm --network nova-id-ory-internal curlimages/curl:latest -s \
  "http://keto:4466/relation-tuples?namespace=Platform" | jq .

# By user
USER_ID="<kratos-identity-id>"
docker run --rm --network nova-id-ory-internal curlimages/curl:latest -s \
  "http://keto:4466/relation-tuples?subject_id=user:${USER_ID}" | jq .

Testing

Stack test

./scripts/test-stack-comprehensive.sh

Covers Kratos, Keto, Hydra, Oathkeeper, and permission layout.

Email

# Test email sending
./scripts/test-email-sending.sh

# With real user (from Kratos)
./scripts/test-email-with-real-user.sh

Inbox: Mailpit at http://localhost:8025.

OAuth2 / Hydra

./scripts/setup-hydra-test-client.sh

Then use the Auth UI or Hydra flows to obtain tokens and call APIs.

Manual API checks

# Health
curl -s http://localhost:8080/health | jq .

# Public
curl -s http://localhost:4455/api/public | jq .

# Kratos identities (admin; ensure 4434 exposed)
curl -s http://localhost:4434/admin/identities | jq '.[] | {id, email: .traits.email, role: .traits.role}'

Application URLs

App URL
Auth UI http://localhost:5173
Admin http://localhost:5174
Test app http://localhost:5175
Mailpit http://localhost:8025
API (gateway) http://localhost:4455

Troubleshooting

Services not healthy

  1. Check docker compose logs for the failing service (often kratos, hydra, keto, or api).

  2. Ensure PostgreSQL is up and migrations have run (kratos-migrate, hydra-migrate, keto-migrate).

  3. Try a full restart:

    docker compose down && docker compose up -d

    Wait 1–2 minutes, then re-check health.

“Expected exactly one rule” (Oathkeeper)

The request matches more than one Oathkeeper access rule. Check config/oathkeeper/rules.local.json / config/oathkeeper/rules.production.json: URL patterns must not overlap (e.g. /api/health vs /api/*). Public routes should be more specific or listed first.

401 on protected endpoints

  • Browser: Ensure you’re logged in and the session cookie is sent (same origin or correct Oathkeeper URL). Clear cookies and log in again.
  • curl: Use -b "ory_kratos_session=..." with a valid session cookie obtained from the browser.

403 on admin endpoints

User must have platform_admin. Assign it:

./scripts/assign-platform-admin-to-user.sh user@example.com

CORS errors

Frontend must call the API via the same origin (e.g. Vite proxy to Oathkeeper) or Oathkeeper must allow the frontend origin. Check vite.config.js proxy and Oathkeeper CORS/config.

Email not received

  • Development: check Mailpit (http://localhost:8025).
  • Verify Kratos courier config in config/kratos/kratos.local.yml (SMTP / Mailpit).
  • Run ./scripts/test-email-sending.sh to confirm sending.

Database issues

# PostgreSQL ready
docker compose exec postgres pg_isready -U postgres

# Connect
docker compose exec postgres psql -U postgres -c "\l"

If you use custom DB names/users, adjust accordingly. For a clean slate, docker compose down -v and then up -d (this removes volumes).

Makefile targets

From project root:

make help              # List targets
make dev               # Start dev stack (docker compose up -d)
make dev-local         # Start dev with local domains + local-proxy (docker-compose.local.yml)
make prod              # Start production stack (docker-compose.production.yml)
make stop              # Stop dev stack
make stop-prod         # Stop production stack
make logs              # Follow logs (dev)
make ps                # Service status (dev)
make health            # Basic health checks
make setup-permissions # Run setup-all-permissions.sh
make clear-permissions # Run clear-all-permissions.sh
make test              # Run test-stack-comprehensive.sh
make test-email        # Run test-email-sending.sh
make test-email-real   # Run test-email-with-real-user.sh
make setup-hydra-client # Run setup-hydra-test-client.sh
make generate-env      # Run generate-env.sh

OAuth back-button (app login)

When a user logs in from the app (OAuth) and then uses the browser Back button, they may land on auth URLs with a stale return_to=...hydra-callback.... The frontend (auth) handles this:

  • HydraCallback and Consent: on stale/used challenge, redirect to the app.
  • Login (session_already_available or no return_to): redirect to the app (never to admin).

The local-proxy (nginx) in config/nginx/local-proxy.conf is only for exposing local domains in development; it is not modified for this behavior. In production with Traefik, if you want to avoid the user seeing a 400 when they Back into GET /api/self-service/login/browser?return_to=...hydra-callback..., you can add a redirect rule in Traefik (e.g. when Sec-Fetch-Mode: navigate and return_to contains hydra-callback, respond with 302 to the app). Otherwise, the frontend redirects from HydraCallback/Consent/Login are enough for most cases.

Production CD Runbook (B3)

Deploys to the production host use a registry-first pull model. Images are pre-built by GitHub Actions on tag push; the server only pulls and restarts containers — it never runs docker build.

Connection details (host, SSH port, deploy user) are stored in GitHub Actions secrets POLARIS2_HOST, POLARIS2_PORT, and POLARIS2_USER. The deploy workflow SSHes in automatically — no manual SSH needed for routine releases.

Normal deploy (tag-triggered)

Push a version tag from main:

git checkout main
git pull
git tag v1.2.0
git push origin v1.2.0

The tag push triggers release.yml only. Deploy is chained after the build, not triggered independently, so the two can never race:

  1. release.yml runs 5 parallel jobs — one per image — each building, Trivy-scanning (fail on CRITICAL/HIGH), and pushing the immutable version tag + :latest to Docker Hub.
  2. When all 5 build jobs succeed, release.yml's final deploy job calls deploy.yml as a reusable workflow (needs: the 5 builds + uses:).
  3. deploy.yml SSHes into the production host and runs scripts/deploy-prod.sh v1.2.0.

Because the deploy job needs all 5 builds, deploy runs only after every image is built and pushed — deploy-prod.sh can safely docker pull the tag with no build/pull race.

The deploy script: pulls images → docker compose up -d --no-build → restarts Oathkeeper → smoke-tests 4 URLs → writes .deploy-version. On smoke failure: auto-rollback to the previous version.

Manual rollback / redeploy

Run deploy.yml directly (it never auto-runs on tag push — it only runs as release.yml's chained job or via this manual dispatch).

Option A — workflow_dispatch (preferred; leaves an audit trail in GitHub):

  1. Go to Actions → Deploy to Production → Run workflow
  2. Enter the target tag (e.g. v1.1.0) in the ref field. The images for that tag must already exist on Docker Hub (they do for any previously released tag).

Option B — direct SSH (emergency only, requires the deploy key):

ssh -i ~/.ssh/nova_id_deploy -p <POLARIS2_PORT> <POLARIS2_USER>@<POLARIS2_HOST> v1.1.0

The SSH command= restriction on the server only allows running deploy-prod.sh; the version tag is passed as SSH_ORIGINAL_COMMAND.

Check what version is running

SSH into the production host and inspect:

# In ~/deploy/nova-id-deploy
cat .deploy-version
docker compose --env-file .env.production -f docker-compose.production.yml ps

Smoke test URLs

All four must return HTTP 200 for a deploy to be considered healthy:

URL Service
https://id.cativo.dev/health/alive Oathkeeper gateway alive
https://id.cativo.dev/.well-known/openid-configuration OIDC metadata
https://id.cativo.dev/api/health BFF (NestJS) health
https://app.cativo.dev/api-test/health demo-api health

First-time bootstrap (one-time only)

After the very first deploy to a new server, Ory databases must be initialised. This is not done by deploy-prod.sh — use:

# On the production host, in ~/deploy/nova-id-deploy
bash scripts/bootstrap-prod.sh --first-boot
bash scripts/deploy-prod.sh v1.0.0
bash scripts/bootstrap-prod.sh --assign-admin carlos@example.com

JWKS rotation (planned maintenance window required)

The Oathkeeper id_token signing key lives at config/oathkeeper/id_token.jwks.json. Rotating it invalidates all active sessions. Never rotate on a routine deploy.

  1. Plan a maintenance window and notify users.
  2. On the production host: bash scripts/generate-jwks.sh (generates new key).
  3. chmod 644 config/oathkeeper/id_token.jwks.json
  4. Push the new JWKS file in a tagged release and deploy normally.
  5. All logged-in users will be logged out — expected.

deploy-prod.sh aborts if JWKS is missing — it never auto-generates it.

Secret rotation (Docker Hub PAT)

# After creating a new PAT at hub.docker.com:
gh secret set DOCKERHUB_TOKEN --body "<new-token>"

Next steps