Add exponential backoff retry for retryable SQLite loAdd exponential backoff retry for retryable SQLite lock errorsck errors

[Wolfman56]

Problem

When fetch_orchestration_item() encounters SQLITE_BUSY errors due to concurrent INSERT/SELECT operations during high-concurrency startup, workflows fail with "database is locked" errors even though PRAGMA busy_timeout = 60000 is configured. This race condition occurs when start_orchestration() INSERTs and the dispatcher tries to SELECT simultaneously.

Solution

Implemented exponential backoff retry logic in the orchestration dispatcher specifically for retryable errors (database locks, connection errors).

Changes

• Track consecutive retryable errors per worker thread
• Exponential backoff delays: 200ms, 400ms, 800ms, 1600ms, 3200ms (capped at 5 seconds)
• Reset counter on successful fetch or permanent errors
• Enhanced error logging showing attempt number and backoff duration

Testing

Tested with concurrent workflow submission scenario:

• ✅ First attempt: Hit database lock (SQLITE_BUSY code 5)
• ✅ Log: Error fetching orchestration item (retryable, attempt 1)...backing off 200ms
• ✅ Second attempt: Succeeded after 200ms backoff
• ✅ Workflow persisted to database and executed successfully
• ✅ Verified in SQLite: Instance created, execution status "Completed"

Impact

• Files changed: 1 file, 24 insertions, 4 deletions
• Backward compatible: Yes, existing behavior preserved for non-retryable errors
• Performance: Minimal overhead (single u32 counter per worker)

This fix resolves the race condition without requiring changes to SQLite pragma configuration or connection pool settings.

[affandar]

Thanks @Wolfman56. Would you pls mind doing the same for the work item dispatcher as well for completeness and symmetry?

[Wolfman56]

I'll have my agent go back and do that! Best.

[Wolfman56]

Ok, dispatcher updated as well

[affandar]

Thanks for the update. Two suggestions below:

1. Simplify the cap logic

let backoff_ms = std::cmp::min(
    100 * (2_u64.pow(consecutive_retryable_errors.min(5))),
    5000  // ← Dead code
);

The exponent is capped at 5, so the effective max is 100 × 2^5 = 3200ms. The .min(5000) can never be reached. 3.2s sounds reasonable—suggest simplifying to an explicit 3s cap:

let backoff_ms = (100 * 2_u64.pow(consecutive_retryable_errors)).min(3000);

2. Worker dispatcher: backoff should be in the main loop

In worker.rs, the sleep happens inside process_next_work_item(), but in orchestration.rs it's inline in the loop. This creates an awkward (bool, bool) return type and splits state management between caller and callee.

Simpler approach—return Result<bool, ProviderError> and let the loop own timing:

async fn process_next_work_item(...) -> Result<bool, ProviderError> {
    match fetch_result {
        Ok(Some(...)) => { /* process */; Ok(true) }
        Ok(None) => Ok(false),
        Err(e) => Err(e),  // Let caller handle
    }
}

// In loop:
match process_next_work_item(...).await {
    Ok(found) => {
        backoff_errors = 0;
        work_found = found;
    }
    Err(e) if e.is_retryable() => {
        backoff_errors += 1;
        let delay_ms = (100 * 2_u64.pow(backoff_errors)).min(3000);
        warn!(attempt = backoff_errors, delay_ms, error = ?e, "Retryable fetch error");
        tokio::time::sleep(Duration::from_millis(delay_ms)).await;
        continue;
    }
    Err(e) => {
        warn!(error = ?e, "Permanent fetch error");
        backoff_errors = 0;
        tokio::time::sleep(Duration::from_millis(100)).await;
        continue;
    }
}

[affandar]

merged, will implement the recommendation I had on top of it in a bit.

[Wolfman56]

Excellent. Sorry got distracted by day job. Keep up the great work.