Deployment and Infrastructure

Deployment & Infrastructure

SecondLayer uses Docker Compose for containerization and a CI/CD pipeline with blue-green deployments to production. The platform runs in two environments: local development and production (there is no staging environment).

Environments

Environment	URL	Deploy Method
Local	`https://local.legal.org.ua`	`docker compose -f docker-compose.local.yml --env-file .env.local up -d`
Production	`https://legal.org.ua`	CI/CD: merge PR to main --> self-hosted runner --> blue-green deploy via SSH

Important: There is no staging environment. All testing happens locally before merging to main.

CI/CD Pipeline

Two GitHub Actions workflows run on a self-hosted runner (local.legal.org.ua):

1. CI/CD: Local Build & Test (`ci-local-deploy.yml`)

Triggered on push to main. Runs on runner labels [self-hosted, local] and [self-hosted, local-build].

Jobs:

Job	Runner	Purpose
Detect Changes	`local`	Identifies which services (backend, rada, openreyestr, frontend, platform, monitoring) changed
Build & Test on Local	`local-build`	Builds shared package, then each changed service; runs unit tests
Self-Heal Build Failures	`local`	On failure, Claude Code agent diagnoses the issue and creates an autofix PR
Deploy to Local & Integration Test	`local`	Builds Docker images, runs migrations, starts local containers, verifies health

2. Deploy to Production (`deploy-prod.yml`)

Triggered after successful Local CI or manually via workflow_dispatch. Runs on [self-hosted, local].

Jobs:

Job	Purpose
Detect Changes	Auto-detect from last deploy marker, or accept manual service list
Pre-Deploy Tests	Build and test changed services before touching production
Self-Heal Test Failures	Claude Code agent creates fix PRs on test failure
Deploy Preview	Phase 1: build inactive color on prod, run migrations, start preview
Promote to Production	Phase 2: switch Nginx upstreams to new color, force-recreate Nginx
Self-Heal Deploy Failures	Claude Code agent diagnoses prod issues and creates fix PRs
Create Release	Tags releases (backend: `v`, frontend: `fe-v`), creates GitHub Releases with changelogs

Deployment Flow

PR merged to main
  --> ci-local-deploy.yml (build, test, deploy locally)
    --> deploy-prod.yml (auto-triggered on CI success)
      --> Phase 1: Build inactive color, run migrations, start preview
      --> Phase 2: Switch Nginx to new color (requires "production" environment approval)
      --> Tag release + deploy marker

Blue-Green Deployment

Production uses blue-green deployment to achieve zero-downtime releases. Colors are tracked per service group (backend and frontend independently):

.active-colors file on the prod server stores current state:
```
backend=blue
frontend=green
```
On deploy, the inactive color is rebuilt and started
After health checks pass, Nginx upstreams are rewritten to point to the new color
Nginx is --force-recreated to pick up upstream changes (bind mount inode staleness)
Safety: the pipeline verifies Docker container state matches .active-colors and self-corrects mismatches

Container Naming Convention

Blue containers use the base name; green containers add a -green suffix:

Service	Blue Container	Green Container
Backend	`secondlayer-app-prod`	`secondlayer-app-prod-green`
RADA	`rada-mcp-app-prod`	`rada-mcp-app-prod-green`
OpenReyestr	`openreyestr-app-prod`	`openreyestr-app-prod-green`
Document Service	`document-service-prod`	`document-service-prod-green`
Frontend	`lexwebapp-prod`	`lexwebapp-prod-green`

Docker Configuration

Compose Files

File	Purpose
`docker-compose.local.yml`	Local development with hot-reload
`docker-compose.prod.yml`	Production deployment (blue + green service definitions)
`docker-compose.gateway.yml`	Unified gateway configuration

Important: Compose files have NO env_file: directives. Secrets are injected via shell env substitution. Always pass --env-file:

docker compose -f docker-compose.local.yml --env-file .env.local up -d
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d

Dockerfiles

Dockerfile	Service
`Dockerfile.mono-backend`	mcp_backend
`Dockerfile.mono-rada`	mcp_rada
`Dockerfile.mono-openreyestr`	mcp_openreyestr
`Dockerfile.document-service`	Document processing service
`Dockerfile.terminal-service`	Admin terminal service
`Dockerfile.opendata-sync`	Open data synchronization service
`Dockerfile.edrsr-fulltext-worker`	EDRSR full-text search worker
`Dockerfile.local-scraper`	Web scraper for data import
`Dockerfile.offshore-import`	Offshore data import
`Dockerfile.r-retrieval`	R-based retrieval service

Port Allocation

Service	HTTP	PostgreSQL	Redis	Qdrant
mcp_backend	3000	5432	6379	6333-6334
mcp_rada	3001	5433	6380	6335-6336
mcp_openreyestr	3005	5435	6382	--

Infrastructure Components

Nginx

Nginx serves as the reverse proxy for production, handling:

SSL termination (via Cloudflare)
Request routing to backend services (blue or green upstream based on active color)
Static file serving for the frontend
WebSocket/SSE proxy for streaming

Critical: Nginx must be --force-recreated after ANY upstream/backend change due to bind mount inode staleness.

Cloudflare

DNS and CDN management
SSL certificates
DDoS protection
Credentials stored in .env.cloudflare (Global API Key + Email)

Database Stack

PostgreSQL 15 with PgBouncer for connection pooling (SCRAM-SHA-256 auth, transaction pool mode)
Redis 7 for caching (TTLs: deputies 7d, bills 1d, laws 30d)
Qdrant for vector similarity search
MinIO for object storage

Monitoring

Production includes a monitoring stack:

Prometheus -- metrics collection
Grafana -- dashboards and alerting
cAdvisor -- container resource metrics
Node Exporter -- host-level metrics
PostgreSQL Exporter -- database metrics (backend + openreyestr)
Redis Exporter -- cache metrics

Management Script

deployment/manage-gateway.sh provides convenience commands for all environments:

cd deployment

# Local development
./manage-gateway.sh start local      # Start all local services
./manage-gateway.sh stop local       # Stop local services
./manage-gateway.sh restart local    # Restart local services
./manage-gateway.sh deploy local     # Full rebuild (--no-cache)
./manage-gateway.sh logs local       # View local logs

# Production (executes via SSH)
./manage-gateway.sh start prod       # Start prod services
./manage-gateway.sh stop prod        # Stop prod services
./manage-gateway.sh deploy prod      # Deploy to prod (cached)
./manage-gateway.sh deploy prod --no-cache  # Deploy prod (full rebuild)
./manage-gateway.sh logs prod        # View prod logs

# Gateway management
./manage-gateway.sh gateway start    # Start Nginx gateway
./manage-gateway.sh gateway stop     # Stop Nginx gateway
./manage-gateway.sh gateway restart  # Restart Nginx gateway
./manage-gateway.sh gateway test     # Test Nginx configuration

# Status and health
./manage-gateway.sh status           # Show container status across all envs
./manage-gateway.sh health           # Check health of all services

# Cleanup (use with caution)
./manage-gateway.sh clean local      # Clean local environment data

Note: Production deployments should go through CI/CD (merge PR to main). The manage-gateway.sh deploy prod command exists for emergency use only.

SSH Access

Production SSH: ssh prod (alias configured in ~/.ssh/config)
SSH key: ~/.ssh/secondlayer-prod
User: ubuntu (never root)
Production server: AWS EC2 instance

Release Versioning

The CI/CD pipeline automatically tags releases after successful production deployment:

Tag Pattern	Scope	Example
`v*`	Backend (mcp_backend, mcp_rada, mcp_openreyestr, shared)	`v1.42.0`
`fe-v*`	Frontend (lexwebapp)	`fe-v1.15.0`
`deploy-prod-*`	Deploy marker (used for change detection between deploys)	`deploy-prod-20260517-143022`

GitHub Releases are automatically created with categorized changelogs (features, fixes, other).

Self-Healing

Both CI/CD pipelines include self-healing capabilities:

Build/test failures trigger a Claude Code agent that:
- Fetches error logs from the failed step
- Diagnoses the root cause
- Creates a fix branch and PR automatically
- Guards against infinite loops (won't attempt autofix on autofix branches)
Deploy failures trigger a separate agent that:
- Collects prod diagnostics (container logs, health checks)
- Proposes and implements a fix
- Creates a PR for review

If no automated fix is possible, a GitHub Issue is created with diagnostic details.

Pre-Deploy Checklist

Run tsc -- fix all TypeScript errors
Run tests locally (npm test in relevant service)
Verify correct env vars for target environment
After deploy: check container health and logs for errors
Never manually recreate prod containers -- use the deploy pipeline

Deployment and Infrastructure

Deployment & Infrastructure

Environments

CI/CD Pipeline

1. CI/CD: Local Build & Test (ci-local-deploy.yml)

2. Deploy to Production (deploy-prod.yml)

Deployment Flow

Blue-Green Deployment

Container Naming Convention

Docker Configuration

Compose Files

Dockerfiles

Port Allocation

Infrastructure Components

Nginx

Cloudflare

Database Stack

Monitoring

Management Script

SSH Access

Release Versioning

Self-Healing

Pre-Deploy Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Getting Started

Architecture

Backend

Services

Business Logic

Frontend & Mobile

Operations

Clone this wiki locally

1. CI/CD: Local Build & Test (`ci-local-deploy.yml`)

2. Deploy to Production (`deploy-prod.yml`)