[INFRA] Implement Structured Logging with Correlation IDs and Log Levels

[Queenode]

🟡 Priority: High
Difficulty: Hard
Estimated Effort: 2-3 days
Relevant Files: src/services/observability.service.ts, src/middleware/, src/index.ts
Labels: infrastructure, priority:high, observability

Requirements

1. Structured Logger

   • Replace all console.log() and console.error() calls across the entire codebase with a structured logger
   • Create src/lib/logger.ts using a logging library (e.g., pino or winston)
   • Every log entry must be JSON with:
     • timestamp (ISO 8601)
     • level (DEBUG, INFO, WARN, ERROR, FATAL)
     • message
     • correlationId (unique per request/webhook)
     • service ("kolo-backend")
     • context (optional object with additional data)

2. Correlation ID Middleware

   • Create src/middleware/correlation-id.middleware.ts
   • Generate a UUID v4 for each incoming request
   • If the request includes an X-Correlation-ID header, use that instead (for distributed tracing)
   • Attach the correlation ID to req.correlationId
   • Pass the correlation ID through to all service calls using AsyncLocalStorage (Node.js async_hooks)
   • Include the correlation ID in the response header X-Correlation-ID

3. Log Level Configuration

   • Default log level from LOG_LEVEL env var (default: INFO)
   • In development: DEBUG level, pretty-printed output
   • In production: INFO level, JSON output (for log aggregation)
   • Never log at DEBUG level in production unless explicitly enabled

4. Sensitive Data Redaction

   • Automatically redact these fields from all log output:
     • stellarWallet, encryptedSecret, iv, authTag, secret, password, token
   • Redact phone numbers: show only last 4 digits (****5678)
   • Redact Stellar public keys: show only first 4 and last 4 characters (GABCD...WXYZ)

5. Performance Logging

   • Log request duration for every HTTP request: { method, path, statusCode, durationMs }
   • Log Stellar/Horizon API call latency: { endpoint, durationMs, statusCode }
   • Log BullMQ job processing time: { jobName, durationMs, status }

6. Testing

   • Unit test: verify JSON output format matches the schema
   • Unit test: verify sensitive data is redacted
   • Unit test: verify correlation ID is propagated through async operations
   • Grep test: verify zero console.log or console.error calls remain in src/
   • Target: 100% console.* elimination, >90% coverage on logger module

---

[DammyAji]

Hello @Queenode, I’d like to take on this issue.
I can implement the structured logging system with correlation IDs, configurable log levels, sensitive-data redaction, and request/service performance logging using the project’s existing architecture.
I’ll go through the current logging, middleware, and service flow first to understand how the codebase handles requests and background jobs, then add the logger, AsyncLocalStorage correlation tracking, and required middleware. I’ll also replace the remaining console.* usage and add tests for log formatting, redaction, correlation propagation, and coverage.
I'd also be sure to follow all the guidelines outlined in the issue to implement changes and deliver effectively.
I’d appreciate the opportunity to contribute to improving Kolo Backend’s observability and reliability. Thank you.