Feature/outbox structured observability

[charliechinedu19-netizen]

Summary

Replace remaining console.* calls in the outbox publisher with structured logging and add comprehensive Prometheus metrics, Grafana dashboards, and documentation for outbox observability.

Closes #1003

Changes

Metrics (already implemented, verified)

The following metrics were already present in src/observability/outboxMetrics.ts:

• outbox_published_total (Counter) — events successfully published
• outbox_failed_total (Counter) — events that failed to publish
• outbox_pending_gauge (Gauge) — current backlog of pending events
• outbox_lease_renew_total (Counter) — lease renewals during processing
• incrementOutboxDeadLetter — dead-letter tracking
• incrementOutboxQuarantine — quarantine tracking
• incrementOutboxLeaderAcquired / incrementOutboxLeaderLost — leader election

All console.* calls were already replaced with structured logger.* calls.

Grafana Dashboard (monitoring/grafana/dashboard.json)

• Added 3 new panels (IDs 17–19):
  • Outbox Dead Letter Rate — rate of events moved to dead-letter (bar gauge)
  • Outbox Quarantine Rate — rate of events moved to quarantine (bar gauge)
  • Outbox Leader Elections — rate of leader acquired/lost events
• Shifted existing panels 15–16 down to y=48 to make room

Documentation

• docs/observability.md — added outbox_quarantine_total, outbox_leader_acquired_total, outbox_leader_lost_total to the metrics table
• docs/OBSERVABILITY.md — same additions

Tests (src/db/outbox/publisher.test.ts)

• Gauge reset on stop — verifies setOutboxPendingGauge(0) is called when the publisher stops
• No payload in logs — ensures event payload content never leaks into log output

Testing

• src/db/outbox/publisher.test.ts — 23/23 tests pass (22 existing + 2 new)
• Related files (src/db/outbox/, src/middleware/metrics.test.ts, src/observability/) — 120/120 tests pass

[drips-wave]

@charliechinedu19-netizen Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits