Description

The EventListener in /backend/src/services/event-listener.ts uses a busy-wait while loop with a 5-second sleep to poll for Soroban events:

async poll(): Promise<void> {
  while (this.isRunning) {
    await this.fetchAndProcessEvents();
    await sleep(5000);  // busy-wait loop
  }
}

Problems

1. No circuit breaker

If fetchAndProcessEvents() throws after startup, the loop catches the error and continues silently. If the Soroban RPC is down for hours, the loop continues hammering the endpoint every 5 seconds with no backoff.

2. No observable health

There is no way to know from outside the class whether the loop is healthy, how many events were processed, or when the last successful poll was.

3. Memory leak risk

If fetchAndProcessEvents() takes longer than 5 seconds (slow RPC), calls overlap. The next interval starts before the previous one finishes.

Fix

Use exponential backoff on errors

async poll(): Promise<void> {
  let backoffMs = 5000;
  const maxBackoffMs = 300000; // 5 minutes max
  
  while (this.isRunning) {
    try {
      await this.fetchAndProcessEvents();
      this.lastSuccessfulPoll = new Date();
      this.consecutiveErrors = 0;
      backoffMs = 5000; // reset on success
    } catch (error) {
      this.consecutiveErrors++;
      backoffMs = Math.min(backoffMs * 2, maxBackoffMs);
      logger.error('EventListener poll failed', { error, consecutiveErrors: this.consecutiveErrors, nextRetryMs: backoffMs });
    }
    await sleep(backoffMs);
  }
}

Prevent overlapping calls with mutex

let isProcessing = false;

if (!isProcessing) {
  isProcessing = true;
  try {
    await this.fetchAndProcessEvents();
  } finally {
    isProcessing = false;
  }
}

Expose health metrics

getHealth() {
  return {
    isRunning: this.isRunning,
    lastSuccessfulPoll: this.lastSuccessfulPoll,
    consecutiveErrors: this.consecutiveErrors,
    lastProcessedLedger: this.lastProcessedLedger,
  };
}

Add to /api/admin/health endpoint

{ "eventListener": { "status": "healthy", "lastPoll": "2025-03-15T10:00:00Z", "consecutiveErrors": 0 } }

Acceptance Criteria

• Exponential backoff on consecutive errors (max 5 minutes)
• No overlapping poll calls
• Health metrics exposed on admin health endpoint
• Alert triggered after 10 consecutive failures
• Poll interval configurable via env var EVENT_LISTENER_INTERVAL_MS