Skip to content

Monitoring and Observability

Jump to bottom
overthelex edited this page May 17, 2026 · 2 revisions

Monitoring & Observability

Monitoring and Observability within the SecondLayer platform is a multi-layered system designed to provide deep visibility into service health, AI model performance, and infrastructure stability. The system integrates real-time metrics collection, structured logging, and automated alerting to ensure the reliability of legal document analysis and MCP tool execution.

The observability stack utilizes Prometheus for metrics ingestion, Grafana for visualization, cAdvisor for container metrics, and Winston-based structured logging for application-level event tracking. This infrastructure allows developers and operators to monitor critical paths such as LLM execution costs, API response latencies, upload pipeline throughput, and database connectivity across both deployment environments (Local and Prod).

Note: There is no staging environment. Only Local and Prod exist.

Core Architecture

The observability architecture follows a standardized pattern across all microservices, including mcp_backend, mcp_rada, and mcp_openreyestr. Each service exposes a /metrics endpoint via prom-client, which Prometheus scrapes at 10-15 second intervals.

flowchart TD
    subgraph Services ["Application Layer"]
        B[mcp_backend :3000]
        R[mcp_rada :3001]
        O[mcp_openreyestr :3005]
    end

    subgraph Collection ["Metrics & Logging"]
        MS[MetricsService<br/>prom-client]
        LOG[Winston Logger]
    end

    subgraph Monitoring ["Infrastructure Layer"]
        PROM[Prometheus v2.48]
        GRAF[Grafana v10.2]
        CA[cAdvisor v0.55]
    end

    subgraph Exporters_List ["Standard Exporters"]
        PE[Postgres Exporter v0.15]
        RE[Redis Exporter v1.56]
        NE[Node Exporter v1.7]
    end

    B & R & O --> MS
    B & R & O --> LOG
    MS --> PROM
    PE & RE & NE --> PROM
    CA --> PROM
    PROM --> GRAF
Loading

Prometheus Configuration

Prometheus is deployed per-environment with separate configuration files:

Environment Config File Retention Port
Local prometheus-local.yml 7 days 127.0.0.1:9090
Prod prometheus-prod.yml 30 days 127.0.0.1:9091

Both share the same rule files (deployment/prometheus/rules/). The --web.enable-lifecycle flag is enabled, allowing hot-reloads of config via /-/reload.

Scrape Targets

All Prometheus configs scrape the following jobs:

Job Interval Target
mcp_backend 10s app:3000
mcp_rada 10s rada-mcp-app:3001
mcp_openreyestr 10s app-openreyestr:3005
postgres_backend 15s postgres-exporter-backend:9187
postgres_openreyestr 15s postgres-exporter-openreyestr:9187
redis 15s redis-exporter:9121
node 15s node-exporter:9100
minio 30s minio:9000 (/minio/v2/metrics/cluster)
qdrant 30s qdrant:6333 (/metrics)
cadvisor 15s cadvisor:8080
prometheus 15s localhost:9090

The local config omits postgres_backend, postgres_openreyestr, and node exporters (lighter footprint for development).

Metrics Collection (MetricsService)

The MetricsService (mcp_backend/src/services/metrics-service.ts) is the primary interface for application-level telemetry. It uses prom-client and collects default Node.js runtime metrics automatically (heap, GC, event loop lag).

Application Metrics

Metric Name Type Labels Description
http_requests_total Counter method, route, status_code Total HTTP requests
http_request_duration_seconds Histogram method, route, status_code Request latency (buckets: 5ms-10s)
pg_pool_connections Gauge state (active/idle/waiting) PostgreSQL pool status
bullmq_jobs Gauge status (waiting/active/completed/failed/delayed) Upload queue job counts
upload_queue_depth Gauge -- Combined waiting+active upload jobs
upload_processing_duration_seconds Histogram status Upload processing time (buckets: 1s-300s)
external_api_calls_total Counter service, status External API calls (openai, anthropic, rada, diia)
external_api_duration_seconds Histogram service External API latency
cost_tracking_total_usd Counter tool_name Cumulative AI cost in USD
sse_active_connections Gauge type (user_stream/message_stream) Active SSE connections
edrsr_cache_operations_total Counter operation, result (hit/miss) EDRSR cache hit/miss
edrsr_vectorizer_docs_processed_total Counter -- Vectorizer documents processed
edrsr_vectorizer_errors_total Counter -- Vectorizer batch errors
edrsr_vectorizer_status Gauge -- Vectorizer state (1=running, 0=stopped, -1=paused)
consultation_bus_messages_total Counter channel Consultation message bus activity
chat_tool_group_requests_total Counter groups LLM meta-tool group requests
cpu_adaptive_concurrency Gauge -- Current BullMQ worker concurrency
cpu_load_average Gauge -- 1-minute CPU load average

Route Normalization

The MetricsService normalizes routes to prevent high-cardinality label explosion:

  • /api/tools/search_court_decisions becomes /api/tools/:toolName
  • UUIDs become :uuid
  • Upload, matter, conversation, and admin IDs are all normalized

Default Node.js Metrics (collected automatically)

  • nodejs_heap_size_used_bytes / nodejs_heap_size_total_bytes
  • nodejs_eventloop_lag_p99_seconds
  • nodejs_gc_duration_seconds_sum
  • nodejs_active_handles_total / nodejs_active_requests_total

Recording Rules

Recording rules (deployment/prometheus/rules/recording-rules.yml) pre-compute expensive queries for dashboard performance:

HTTP Aggregations (15s interval)

Record Expression
service:http_requests:rate5m Request rate per service
service:http_errors:rate5m 5xx error rate per service
service:http_error_rate:ratio5m Error rate as percentage
service:http_request_duration:p50 P50 latency per service
service:http_request_duration:p95 P95 latency per service
service:http_request_duration:p99 P99 latency per service

Cost Aggregations (60s interval)

Record Expression
cost:total_usd:rate1h Total cost per hour (USD)
cost:per_tool_usd:rate1h Cost per tool per hour

PG Pool Aggregations (15s interval)

Record Expression
pg:pool_utilization:ratio active / (active + idle)

Node.js Runtime Aggregations (15s interval)

Record Expression
nodejs:heap_utilization:ratio Heap used / total
nodejs:gc_time:ratio5m GC time as fraction of wall-clock

Alerting

Alert Rules

Alert rules are defined in deployment/prometheus/rules/alert-rules.yml and organized into two groups:

Critical Alerts

Alert Condition Duration Description
ServiceDown up == 0 1m Any scrape target is unreachable
PgPoolExhausted pg_pool_connections{state="waiting"} > 10 2m PostgreSQL pool has 10+ waiting connections
RedisMemoryCritical Redis memory > 95% of max 5m Redis approaching OOM
RedisEvictedKeys Eviction rate > 10/s 5m Redis evicting keys under pressure
PgCacheHitLow Cache hit ratio < 95% 10m PostgreSQL needs more shared_buffers

Warning Alerts

Alert Condition Duration Description
HighErrorRate 5xx rate > 5% 5m Service returning too many errors
HighLatency P95 > 5s 5m Slow response times
PgPoolHigh Pool utilization > 80% 5m Connection pool nearing capacity
PgIdleInTransaction > 10 idle-in-transaction 5m Leaked transactions
UploadQueueHigh Queue depth > 100 10m Upload processing backlog
CostSpike Hourly cost > 3x 24h average 10m Unusual AI spend
HighCpuIowait CPU iowait > 30% 10m Disk I/O bottleneck
HighMemoryUsage System memory > 80% 10m Host running low on RAM
HighEventLoopLag P99 event loop lag > 0.5s 5m Blocking operations in Node.js
HeapMemoryHigh Heap > 90% utilized 10m Possible memory leak

Alert Routing (Grafana)

Alert notifications are configured via Grafana provisioning (deployment/grafana/provisioning/alerting/contact-points.yml):

  • Contact point: email-admin sends to grafana@legal.org.ua
  • Group by: alertname + severity
  • Timing: 30s group wait, 5m group interval, 4h repeat interval

Grafana Dashboards

Six pre-provisioned dashboards are deployed from deployment/grafana/dashboards/:

# Dashboard Key Panels
01 System Overview Total RPS, Error Rate %, P95 Latency, Cost/Hour, Service Health (backend/rada/openreyestr), PG Pool, Redis Memory, Upload Queue Depth
02 Backend Detail Request Rate by Route, Latency Histogram (P50/P95/P99), Status Code Distribution, PG Pool Connections (Stacked), External API Calls and Duration
03 Upload Pipeline BullMQ Jobs by Status, Processing Duration (P50/P95/P99), Queue Depth Trend, Active vs Max Concurrency
04 API Costs Cost/Hour, Cost/Day, Cost/Month (extrapolated), Cost Trend by Tool, Top 10 Expensive Tools
05 Infrastructure PG Active Connections, Transaction Rate, Cache Hit Ratio, Redis Memory/Clients/Commands/Evictions, CPU by Mode, Memory, Disk I/O, Network Traffic, Event Loop Lag, Heap Memory, GC Duration, Active Handles
06 External Data Sources NAIS Registry record counts (12 registries), PostgreSQL stats for both backend and openreyestr DBs, RADA table counts (deputies/bills/committees/factions/voting), Backend table counts (documents/users/EDRSR), Redis details, Qdrant vectors/collections, MinIO storage/objects, Service Health

Grafana Configuration

  • Datasource: Prometheus (auto-provisioned, URL resolves via $PROMETHEUS_HOST)
  • Dashboard provider: Loads JSON files from /var/lib/grafana/dashboards (bind-mounted from deployment/grafana/dashboards/)
  • Access:
    • Prod: https://grafana.legal.org.ua (port 3101 internally, proxied by nginx)
    • Local: https://local.grafana.legal.org.ua (anonymous viewer enabled for dev)

Structured Logging

The system uses Winston-based structured logging from packages/shared/src/utils/logger.ts, shared across all services.

Configuration

createLogger(serviceName: string)
  • Format: JSON with timestamp and stack traces on errors
  • Default meta: { service: serviceName }
  • Log level: Controlled by LOG_LEVEL env var (default: info)

Transport Behavior

Condition Transports
Default Console (colorized) + File (logs/error.log, logs/combined.log)
MCP_STDIO_MODE=true File only (stdout reserved for MCP protocol)
LOG_TO_FILE=false Console only (used in Docker where docker logs captures stdout/stderr)

Console Output Routing

  • error and warn go to stderr
  • info and debug go to stdout

This allows Docker log drivers to capture both streams and enables filtering by severity in log aggregation.

Docker Log Access

Since services run in containers, logs are accessed via:

# Tail live logs
docker compose -f docker-compose.prod.yml logs -f mcp-backend-prod

# Last 100 lines
docker compose -f docker-compose.prod.yml logs --tail=100 mcp-backend-prod

Infrastructure Exporters

Exporter Image Port Targets
Postgres Exporter (backend) prometheuscommunity/postgres-exporter:v0.15.0 9187 secondlayer_prod DB
Postgres Exporter (openreyestr) prometheuscommunity/postgres-exporter:v0.15.0 9187 openreyestr_prod DB
Redis Exporter oliver006/redis_exporter:v1.56.0 9121 Redis instance
Node Exporter prom/node-exporter:v1.7.0 9100 Host OS (CPU, RAM, Disk, Network)
cAdvisor gcr.io/cadvisor/cadvisor:v0.55.1 8080 Docker containers (runs privileged, docker_only mode)

All exporters are deployed as Docker containers within the same network as the services they monitor. They are accessible only on 127.0.0.1 (not exposed publicly).

Environment Differences

Aspect Local Prod
Prometheus retention 7 days 30 days
Prometheus port 9090 9091
Grafana anonymous access Yes (Viewer role) No
Node Exporter Not deployed Deployed
PostgreSQL Exporters Not deployed Both (backend + openreyestr)
cAdvisor flags Default docker_only=true, 30s housekeeping
Alert notifications Disabled Email to grafana@legal.org.ua

Key Files

Path Purpose
deployment/prometheus/prometheus-local.yml Prometheus config for local
deployment/prometheus/prometheus-prod.yml Prometheus config for production
deployment/prometheus/rules/alert-rules.yml Alerting rules (critical + warning)
deployment/prometheus/rules/recording-rules.yml Pre-computed aggregation rules
deployment/grafana/dashboards/*.json Six Grafana dashboard definitions
deployment/grafana/provisioning/datasources/prometheus.yml Datasource auto-provisioning
deployment/grafana/provisioning/dashboards/provider.yml Dashboard file provider
deployment/grafana/provisioning/alerting/contact-points.yml Alert notification routing
mcp_backend/src/services/metrics-service.ts Application metrics (prom-client)
packages/shared/src/utils/logger.ts Shared Winston logger factory

SecondLayer

Getting Started

Architecture

Backend

Services

Business Logic

Frontend & Mobile

Operations

Clone this wiki locally