Epic: observability, alerts, and recovery runbooks

[osvaldoandrade]

Goal

Make failures visible, diagnosable, and recoverable through metrics, alerts, and documented operational flows.

Background

The observed failure combined unhealthy nodes, stuck pods, stale processes, and CPU consumption without functional service. These states need first-class metrics and runbooks so operators can detect and recover before manual investigation becomes the primary tool.

Outcomes

• Metrics expose quorum, Raft role, epoch, duplicate identity, probe state, repair actions, and stuck lifecycle phases.
• Alerts cover degraded quorum, duplicate identity, stuck terminating pods, NodeNotReady, and failed repairs.
• Runbooks document drain, decommission, fencing, repair, restore, and post-recovery validation.

Child issues

• Add resilience metrics, dashboards, and alerts #588 Add resilience metrics, dashboards, and alerts
• Write recovery runbooks for drain, fencing, repair, and restore #589 Write recovery runbooks for drain, fencing, repair, and restore