diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index fe9a871..541cee3 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -34,7 +34,7 @@ export LOG_LEVEL="info" export NODE_ENV="production" ``` -### `ALLOW_SERVER_SIGNING` — Server-Side Transaction Signing (#596) +### `ALLOW_SERVER_SIGNING` G�� Server-Side Transaction Signing (#596) | Variable | Default | Purpose | | ---------------------- | --------------- | ----------------------------------------------------------------------------------- | @@ -104,7 +104,7 @@ export ALLOW_SERVER_SIGNING=true export STELLAR_SECRET_KEY="S..." export SERVER_SIGNING_API_KEY="$(openssl rand -hex 32)" -# Production (public) — leave unset; users sign via Freighter wallet +# Production (public) G�� leave unset; users sign via Freighter wallet # ALLOW_SERVER_SIGNING is intentionally absent ``` @@ -194,7 +194,7 @@ backend configured by `SECRET_BACKEND`. ```bash export SECRET_BACKEND=env -export KEEPER_SECRET="S..." # .env or shell — never commit +export KEEPER_SECRET="S..." # .env or shell G�� never commit npx ts-node scripts/keeper.ts ``` @@ -221,7 +221,7 @@ environments. ### Backend: `github` (GitHub Actions CI/CD) 1. Add `KEEPER_SECRET` in your repository: - **Settings → Secrets and variables → Actions → New repository secret** + **Settings G�� Secrets and variables G�� Actions G�� New repository secret** 2. Reference in your workflow: ```yaml jobs: @@ -255,7 +255,7 @@ requires `ceil(S / L)` consecutive keeper runs. After the final window is proces the cursor resets to 0 and the next run begins a fresh sweep. ``` -Example: 50 schedule entries, MAINTENANCE_LIMIT=10 → 5 runs for full coverage. +Example: 50 schedule entries, MAINTENANCE_LIMIT=10 G�� 5 runs for full coverage. ``` **Tuning recommendations:** @@ -281,7 +281,7 @@ workflows correctly report success or failure. | Exit Code | Meaning | | --------- | ------- | -| `0` | Keeper completed successfully — all recipients maintained, instance bumped, balance checked. | +| `0` | Keeper completed successfully G�� all recipients maintained, instance bumped, balance checked. | | `1` | Keeper encountered a fatal error (missing config, RPC failure, transaction error, etc.). The alert webhook fires **before** the non-zero exit. | **CI behaviour:** @@ -300,7 +300,7 @@ npx ts-node scripts/keeper.ts; echo "exit: $?" # Failure path (missing CONTRACT_ID) CONTRACT_ID= npx ts-node scripts/keeper.ts; echo "exit: $?" -# → prints exit: 1 +# G�� prints exit: 1 ``` The `main` function is exported for programmatic testing via subprocess @@ -327,7 +327,7 @@ The batch-vesting contract enforces a **single whitelisted fee asset** stored in **Key points:** - The fee asset is set **once** during contract initialization via `set_config()` -- `set_fee_config()` **no longer accepts** a `fee_asset` parameter — it only sets `fee_per_recipient` and `treasury` +- `set_fee_config()` **no longer accepts** a `fee_asset` parameter G�� it only sets `fee_per_recipient` and `treasury` - All deposit fees are automatically collected in the whitelisted asset - Changing the fee asset requires a full `set_config()` call (admin-only) @@ -366,7 +366,7 @@ stellar contract invoke \ "fee_asset": "" }' -# 4. Set fee parameters (fee_asset NOT included — comes from config) +# 4. Set fee parameters (fee_asset NOT included G�� comes from config) stellar contract invoke \ --id \ --source deployer \ @@ -500,13 +500,13 @@ misconfigured. Use it as a readiness probe. | Topology | Mode | Safe? | | --------------------------------- | ------------- | ----- | -| Single container / single process | `single-node` | ✅ | -| Multiple replicas, shared volume | `single-node` | ⚠️ Only with a single writer | -| Multiple replicas, no shared disk | `ha` | ✅ | -| Serverless (ephemeral `/tmp`) | `ha` | ✅ | -| Serverless (ephemeral `/tmp`) | `single-node` | ❌ Data lost on cold start | +| Single container / single process | `single-node` | G�� | +| Multiple replicas, shared volume | `single-node` | G��n+� Only with a single writer | +| Multiple replicas, no shared disk | `ha` | G�� | +| Serverless (ephemeral `/tmp`) | `ha` | G�� | +| Serverless (ephemeral `/tmp`) | `single-node` | G�� Data lost on cold start | -**Health check** — verify store connectivity before routing traffic: +**Health check** G�� verify store connectivity before routing traffic: ```bash curl -s http://localhost:3000/api/health @@ -543,15 +543,45 @@ vercel --prod - Global CDN - Preview deployments - Easy rollback - ### Option 2: Docker Container For flexibility and multi-platform deployment, use the committed [`Dockerfile`](../Dockerfile) at the repo root. It is a multi-stage build -based on `node:22-alpine` that: - -````dockerfile -FROM node:22-alpine +that: + +1. **Builder stage** (`node:22-alpine`): Installs build dependencies for native + modules (Python, make, g++) and runs `npm ci` + `npm run build` +2. **Production stage** (`node:22-alpine`): Copies only production artifacts, + creates a non-root `nodejs` user, and includes a healthcheck + +```dockerfile +# Builder stage +FROM node:22-alpine AS builder +RUN apk add --no-cache python3 make g++ libc-dev +WORKDIR /app +COPY package*.json ./ +RUN npm ci +COPY . . +RUN npm run build + +# Production stage +FROM node:22-alpine AS production +RUN addgroup -S nodejs && adduser -S nodejs -G nodejs +WORKDIR /app +COPY --from=builder /app/node_modules ./node_modules +COPY --from=builder /app/.next/standalone ./ +COPY --from=builder /app/.next/static ./.next/static +COPY --from=builder /app/public ./public +COPY --from=builder /app/lib ./lib +COPY --from=builder /app/scripts ./scripts +RUN mkdir -p /app/data && chown -R nodejs:nodejs /app +USER nodejs +ENV NODE_ENV=production +EXPOSE 3000 +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1 +CMD ["node", "server.js"] +``` The accompanying `.dockerignore` keeps `node_modules`, `.next`, `contracts/target`, `.env*`, and `data/` out of the build context. @@ -559,7 +589,7 @@ The accompanying `.dockerignore` keeps `node_modules`, `.next`, **Build:** ```bash docker build -t stellar-bulk-pay:latest . -```` +``` **Run locally:** @@ -586,468 +616,3 @@ docker push myregistry/stellar-bulk-pay:latest - Azure Container Instances ### Option 3: Traditional VPS - -For complete control: - -```bash -# SSH to server -ssh user@server.com - -# Clone repository -git clone https://github.com/your-org/stellar-bulk-pay.git -cd stellar-bulk-pay - -# Install dependencies -npm ci --only=production - -# Build -npm run build - -# Set environment -export STELLAR_SECRET_KEY="S..." - -# Start with process manager (PM2) -npm install -g pm2 -pm2 start npm --name "stellar-bulk-pay" -- start -pm2 save -pm2 startup -``` - -## Security Considerations - -### 1. Secret Management - -**Never:** - -- Commit `.env` files -- Pass secrets as command-line arguments -- Log secret keys -- Store in comments or documentation - -**Always:** - -- Use environment variables -- Rotate keys regularly -- Use secret management service -- Audit access logs - -### 2. Network Security - -```nginx -# HTTPS configuration -server { - listen 443 ssl http2; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Enforce HTTPS - ssl_protocols TLSv1.2 TLSv1.3; - ssl_ciphers HIGH:!aNULL:!MD5; - - location / { - proxy_pass http://localhost:3000; - proxy_set_header X-Forwarded-Proto $scheme; - } -} -``` - -### 3. Rate Limiting - -Protect against abuse: - -```typescript -// Example rate limiter middleware -import rateLimit from "express-rate-limit"; - -const apiLimiter = rateLimit({ - windowMs: 15 * 60 * 1000, // 15 minutes - max: 100, // Limit each IP to 100 requests per windowMs -}); - -app.use("/api/", apiLimiter); -``` - -The app itself doesn't use the example above — it applies `applyRateLimit()` / -`setRateLimitHeaders()` per-route from `lib/api-rate-limit.ts`, with policies -keyed by endpoint and subscription tier (see `DEFAULT_LIMITS` in that file). -Every rate-limited response carries `X-RateLimit-Limit`, `X-RateLimit-Remaining`, -and `X-RateLimit-Reset` headers, plus `Retry-After` when blocked (HTTP 429). - -Current per-endpoint policies (requests per rolling window, per tier): - -| Endpoint | Free | Pro | Enterprise | Window | Why | -| --------------------- | ---- | --- | ---------- | ------ | ---------------------------------------------------------------- | -| `batch-build` | 8 | 20 | 60 | 60s | Builds unsigned transactions | -| `batch-submit` | 5 | 15 | 45 | 60s | Enqueues paid, server-signed work | -| `batch-submit-signed` | 5 | 15 | 45 | 60s | Enqueues paid, client-signed work | -| `batch-retry` | 5 | 15 | 45 | 60s | Can re-enqueue paid, server-signed work (#743) | -| `batch-recover` | 30 | 100 | 300 | 60s | Enumerable per-job detail lookup (#743) | -| `batch-history` | 20 | 60 | 180 | 60s | Enumerable, supports search/aggregation across all jobs (#743) | -| `batch-status` | 60 | 200 | 600 | 60s | Lightweight single-job polling | -| `batch-events` | 10 | 30 | 90 | 60s | SSE stream open | -| `tx-status` | 30 | 100 | 300 | 60s | Single transaction status lookup | -| `dashboard-metrics` | 20 | 60 | 180 | 60s | Aggregation across jobs | -| `webhook-register` | 3 | 10 | 30 | 60s | Registers outbound webhook callbacks | -| `health` | 30 | 100 | 300 | 60s | Liveness/readiness checks | - -`RATE_LIMIT_BACKEND` (see above) determines where counters are stored — -SQLite for single-instance deployments, Redis or Postgres for HA. Limits can -be tuned via environment variables read by `tunedLimit()` in -`lib/api-rate-limit.ts` without redeploying code. - -### 4. Input Validation - -Always validate at the edge: - -```typescript -// Validate batch size -if (payments.length > 10000) { - return NextResponse.json( - { error: "Batch size exceeds limit" }, - { status: 400 }, - ); -} -``` - -### 5. Logging Security - -**Safe to log:** - -- Transaction hashes -- Public keys (anonymized) -- Error types (not messages) -- Timestamps - -**Never log:** - -- Secret keys -- Full request/response bodies -- User IP addresses (unless authorized) -- Sensitive amounts - -```typescript -// Safe logging -console.log("[Payment] Transaction submitted:", { - hash: txHash, - recipientCount: payments.length, - timestamp: new Date().toISOString(), -}); - -// Avoid -console.log("[Payment] Full config:", config); // Might contain secrets -``` - -## Monitoring and Observability - -### Application Metrics - -Track key metrics: - -```typescript -// Example with StatsD -import StatsD from "node-dogstatsd"; - -const dogstatsd = new StatsD(); - -// Track batch submissions -dogstatsd.gauge("batches.size", payments.length); -dogstatsd.timing("batches.duration", duration); -dogstatsd.increment("batches.successful"); -dogstatsd.increment("batches.failed"); -``` - -### Error Tracking - -Use a service like Sentry: - -```typescript -import * as Sentry from "@sentry/nextjs"; - -Sentry.init({ - dsn: process.env.SENTRY_DSN, - tracesSampleRate: 1.0, -}); - -// Errors are automatically captured -``` - -### Log Aggregation & Structured Request Logging - -The application includes a structured JSON logger located at `lib/logger.ts` and Next.js middleware that assigns a unique correlation ID (`x-request-id`) to every incoming API request. The logger automatically anonymizes sensitive Stellar public keys (e.g., truncating them to `GB3...XYZ`) and outputs logs in JSON format: - -```json -{ - "level": "info", - "timestamp": "2026-05-31T20:00:00.000Z", - "requestId": "a4f9c8f0-1e0f-4d77-9db6-9afcd21b8d05", - "jobId": "5f8b3c20-3b02-4e63-bd4f-3f6291a13bfd", - "publicKey": "GB3...XYZ", - "network": "testnet", - "msg": "Batch submit job queued and background worker triggered" -} -``` - -#### Datadog / CloudWatch Integration - -1. **Datadog log ingestion**: - - Ensure the Next.js runtime environment sends stdout/stderr logs directly. - - In Datadog log configuration, enable the JSON parser so fields like `level`, `requestId`, and `jobId` are automatically parsed into searchable attributes. - - Configure a mapping for standard attributes: map `level` to status, `timestamp` to date, and `msg` to message. - -2. **AWS CloudWatch**: - - Structured JSON logs are automatically parsed by CloudWatch logs. - - Use CloudWatch Logs Insights to query and trace invocations across serverless instances using `requestId` or `jobId`: - ```sql - fields @timestamp, level, requestId, jobId, msg - | filter requestId = "a4f9c8f0-1e0f-4d77-9db6-9afcd21b8d05" - | sort @timestamp asc - ``` - -## Database Setup (Optional) - -For production batch tracking: - -### PostgreSQL Setup - -```sql -CREATE TABLE batches ( - id SERIAL PRIMARY KEY, - created_at TIMESTAMP NOT NULL DEFAULT NOW(), - submitted_at TIMESTAMP, - network VARCHAR(20) NOT NULL, - total_recipients INTEGER NOT NULL, - total_amount DECIMAL(20, 7) NOT NULL, - transaction_count INTEGER NOT NULL, - successful_count INTEGER, - failed_count INTEGER, - status VARCHAR(20) NOT NULL, - data JSONB NOT NULL -); - -CREATE TABLE payments ( - id SERIAL PRIMARY KEY, - batch_id INTEGER NOT NULL REFERENCES batches(id), - recipient VARCHAR(56) NOT NULL, - amount DECIMAL(20, 7) NOT NULL, - asset VARCHAR(255) NOT NULL, - transaction_hash VARCHAR(64), - status VARCHAR(20) NOT NULL, - error_message TEXT -); - -CREATE INDEX idx_batches_created_at ON batches(created_at); -CREATE INDEX idx_batches_network ON batches(network); -CREATE INDEX idx_payments_batch_id ON payments(batch_id); -``` - -## Performance Optimization - -### Caching - -Cache validator results: - -```typescript -const validationCache = new Map(); - -function validateCached(payment: PaymentInstruction) { - const key = JSON.stringify(payment); - if (validationCache.has(key)) { - return validationCache.get(key); - } - const result = validatePaymentInstruction(payment); - validationCache.set(key, result); - return result; -} -``` - -### Connection Pooling - -For database connections: - -```typescript -import { Pool } from "pg"; - -const pool = new Pool({ - max: 20, - min: 4, - idleTimeoutMillis: 30000, -}); -``` - -### Batch Optimization - -Tune batch size based on network conditions: - -```typescript -// Adaptive batch sizing -const getBatchSize = (network: "testnet" | "mainnet") => { - if (network === "testnet") return 100; - // Mainnet might have higher fees, use smaller batches - return 50; -}; -``` - -## Rollback Plan - -### Version Control - -```bash -# Tag releases -git tag -a v1.0.0 -m "Production release" -git push origin v1.0.0 - -# Easy rollback if needed -git checkout v0.9.0 -npm run build -npm start -``` - -### Blue-Green Deployment - -Maintain two versions: - -```bash -# Deploy new version to "green" environment -npm run deploy:green - -# Test thoroughly -npm run test:e2e - -# Switch traffic -npm run switch:traffic -``` - -## Testnet to Mainnet Migration - -### 1. Validate on Testnet - -```bash -# Test with testnet funds -STELLAR_SECRET_KEY="S..." npm run dev -# Submit test batches -# Verify transaction hashes on stellar.expert -``` - -### 2. Prepare Mainnet Account - -```bash -# Create mainnet account -# Fund with adequate XLM -# Test basic operations - -# Verify account -curl https://horizon.stellar.org/accounts/YOUR_PUBLIC_KEY -``` - -### 3. Gradual Migration - -```bash -# Start with small batches -# Monitor for issues -# Gradually increase batch sizes -# Monitor transaction costs and success rates -``` - -### 4. Monitor Closely - -```bash -# Check account balance -curl https://horizon.stellar.org/accounts/YOUR_PUBLIC_KEY/balances - -# Review transaction history -curl "https://horizon.stellar.org/accounts/YOUR_PUBLIC_KEY/transactions" - -# Monitor for errors -grep "ERROR" application.log -``` - -## Maintenance - -### Regular Tasks - -- **Daily**: Review error logs and transaction status -- **Weekly**: Monitor account balance and transaction costs -- **Monthly**: Review and archive logs, update dependencies -- **Quarterly**: Security audit, performance review - -### Backup Strategy - -```bash -# Backup application logs -tar -czf logs-backup-$(date +%Y%m%d).tar.gz /var/log/stellar-bulk-pay/ - -# Backup database -pg_dump stellar_bulk_pay > backup-$(date +%Y%m%d).sql - -# Store offsite -aws s3 cp logs-backup-*.tar.gz s3://backups/ -``` - -### Updates - -```bash -# Check for updates -npm outdated - -# Update dependencies -npm update - -# Test thoroughly -npm test -npm run build - -# Deploy updated version -git commit -am "Update dependencies" -git push origin main -``` - -## Disaster Recovery - -### Account Recovery - -If secret key is compromised: - -1. Create new Stellar account -2. Transfer remaining funds -3. Update environment variables -4. Reissue all ongoing operations -5. Review transaction history - -### Data Recovery - -```bash -# Restore from backup -psql stellar_bulk_pay < backup-20240101.sql - -# Verify integrity -SELECT COUNT(*) FROM batches; -``` - -### Incident Response - -```bash -# 1. Identify issue -grep ERROR /var/log/stellar-bulk-pay/error.log - -# 2. Stop processing -pm2 stop stellar-bulk-pay - -# 3. Investigate -# Review logs, check Stellar network status - -# 4. Fix and redeploy -git checkout main && npm run build && pm2 start stellar-bulk-pay - -# 5. Verify -curl http://localhost:3000/api/health -``` - -## Support - -For deployment issues: - -- Check application logs -- Review Stellar network status -- Consult DEVELOPMENT.md for debugging -- Open GitHub issue with logs (no secrets) diff --git a/Dockerfile b/Dockerfile index ca65416..4ae57c3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,14 +1,56 @@ -FROM node:22-alpine +# syntax=docker/dockerfile:1.4 + +# ============================================================================= +# Build stage: install dependencies and compile native modules (better-sqlite3) +# ============================================================================= +FROM node:22-alpine AS builder + +# Install build dependencies for native modules +RUN apk add --no-cache \ + python3 \ + make \ + g++ \ + libc-dev WORKDIR /app +# Copy package files and install dependencies COPY package*.json ./ RUN npm ci +# Copy source and build COPY . . RUN npm run build +# ============================================================================= +# Production stage: minimal runtime image with non-root user +# ============================================================================= +FROM node:22-alpine AS production + +# Create group and user for running as non-root +RUN addgroup -g 1001 -S nodejs && \ + adduser -S nodejs -u 1001 -G nodejs + +WORKDIR /app + +# Copy production dependencies and built assets from builder +COPY --from=builder /app/node_modules ./node_modules +COPY --from=builder /app/.next/standalone ./ +COPY --from=builder /app/.next/static ./.next/static +COPY --from=builder /app/public ./public +COPY --from=builder /app/lib ./lib +COPY --from=builder /app/scripts ./scripts + +# Create data directory with correct ownership +RUN mkdir -p /app/data && \ + chown -R nodejs:nodejs /app + +USER nodejs + ENV NODE_ENV=production EXPOSE 3000 -CMD ["npm", "start"] +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1 + +CMD ["node", "server.js"] \ No newline at end of file diff --git a/docker_section.md b/docker_section.md new file mode 100644 index 0000000..2d62ac9 --- /dev/null +++ b/docker_section.md @@ -0,0 +1,73 @@ +### Option 2: Docker Container + +For flexibility and multi-platform deployment, use the committed +[`Dockerfile`](../Dockerfile) at the repo root. It is a multi-stage build +that: + +1. **Builder stage** (`node:22-alpine`): Installs build dependencies for native + modules (Python, make, g++) and runs `npm ci` + `npm run build` +2. **Production stage** (`node:22-alpine`): Copies only production artifacts, + creates a non-root `nodejs` user, and includes a healthcheck + +```dockerfile +# Builder stage +FROM node:22-alpine AS builder +RUN apk add --no-cache python3 make g++ libc-dev +WORKDIR /app +COPY package*.json ./ +RUN npm ci +COPY . . +RUN npm run build + +# Production stage +FROM node:22-alpine AS production +RUN addgroup -S nodejs && adduser -S nodejs -G nodejs +WORKDIR /app +COPY --from=builder /app/node_modules ./node_modules +COPY --from=builder /app/.next/standalone ./ +COPY --from=builder /app/.next/static ./.next/static +COPY --from=builder /app/public ./public +COPY --from=builder /app/lib ./lib +COPY --from=builder /app/scripts ./scripts +RUN mkdir -p /app/data && chown -R nodejs:nodejs /app +USER nodejs +ENV NODE_ENV=production +EXPOSE 3000 +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1 +CMD ["node", "server.js"] +``` + +The accompanying `.dockerignore` keeps `node_modules`, `.next`, +`contracts/target`, `.env*`, and `data/` out of the build context. + +**Build:** +```bash +docker build -t stellar-bulk-pay:latest . +``` + +**Run locally:** + +```bash +docker run --rm -p 3000:3000 \ + -e NODE_ENV=production \ + -e STELLAR_SECRET_KEY="$STELLAR_SECRET_KEY" \ + -v "$(pwd)/data:/app/data" \ + stellar-bulk-pay:latest +``` + +**Push:** + +```bash +docker tag stellar-bulk-pay:latest myregistry/stellar-bulk-pay:latest +docker push myregistry/stellar-bulk-pay:latest +``` + +**Deploy to container service:** + +- AWS ECS — mount an EFS volume at `/app/data` if you need durable SQLite. +- Google Cloud Run — pair with a managed database, or accept that + `data/` resets on each container instance. +- Azure Container Instances + +### Option 3: Traditional VPS \ No newline at end of file