Skip to content

fix: enable KillSwitchService auto-recovery by default with circuit breaker (#373) - #377

Merged
akordavid373 merged 1 commit into
connect-boiz:mainfrom
graceful-tech12:fix/issue-373-killswitch-auto-recovery
Jul 29, 2026
Merged

fix: enable KillSwitchService auto-recovery by default with circuit breaker (#373)#377
akordavid373 merged 1 commit into
connect-boiz:mainfrom
graceful-tech12:fix/issue-373-killswitch-auto-recovery

Conversation

@graceful-tech12

Copy link
Copy Markdown
Contributor

Summary

Fixes #373 — KillSwitchService Auto-Recovery Disabled by Default

Problem

The HSM integration was initialized with enableAutoRecovery: false, meaning when the KillSwitchService activated, the system entered a permanently locked-down state requiring manual admin intervention.

Changes

  • Enable auto-recovery by default with 5-minute initial delay and exponential backoff
  • Circuit-breaker probe pattern: health check before full deactivation; if probe fails, double backoff and retry
  • Source-based recovery delays: HSM transient failure → 30s, system failure → 2min, security incident → 5min
  • Max recovery attempt limit (default: 5) with escalation to human operators when exhausted
  • Prometheus gauge kill_switch_recovery_attempts for monitoring
  • 6 new test cases for auto-recovery scenarios
  • Fixed pre-existing duplicate metric name collisions in prometheus.ts

Files Changed

File Lines
backend/src/services/killSwitchService.ts +240/-30
backend/src/services/hsmIntegration.ts +7/-2
backend/src/index.ts +3/-1
backend/src/utils/prometheus.ts +13/-2
backend/src/tests/hsm.integration.test.ts +165/-4

Test Results

247/247 tests pass across all 16 test suites (zero regressions)

Acceptance Criteria

  • Enable auto-recovery by default with reasonable delay + exponential backoff
  • Max recovery attempt limit (5) with escalation to human operators
  • Circuit breaker pattern: probe → succeed → restore; fail → double backoff
  • Separate HSM transient failure trigger (30s) vs security incident (5min+)
  • Test: threshold breach → auto-recovery → successful recovery → normal operation
  • Test: HSM failure → shorter recovery delay than security incident
  • Prometheus gauge kill_switch_recovery_attempts

…reaker (connect-boiz#373)

- Enable auto-recovery by default (was disabled) with 5-minute initial delay
- Add circuit-breaker probe pattern: health check before full deactivation
- Implement source-based recovery delays (HSM failure: 30s, security: 5min)
- Add max recovery attempt limit (default: 5) with escalation to operators
- Add kill_switch_recovery_attempts Prometheus gauge
- Add 6 new test cases for auto-recovery scenarios
- Fix duplicate metric name collisions in prometheus.ts

Fixes: connect-boiz#373
Assigned-to: graceful-tech12
@akordavid373
akordavid373 merged commit 79e5374 into connect-boiz:main Jul 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[HIGH] KillSwitchService Auto-Recovery Disabled by Default — Permanent System Lockout Risk

2 participants