Skip to content

core: background job queue for chain submission and long-running work #54

Description

@joelpeace48-cell

Problem

Submitting a Soroban transaction and waiting for confirmation takes several
seconds. Doing that inside a request handler ties up a connection, times out
under load, and loses the work entirely if the process restarts mid-flight.

What to do

  • A durable job queue — BullMQ on Redis, or a Postgres-backed queue if the
    dependency footprint matters more than throughput.
  • Jobs for: transaction submission and confirmation polling, export generation,
    indexer catch-up, webhook fan-out.
  • Retry with backoff, a bounded attempt count, and a dead-letter queue that a
    human can inspect.
  • Idempotent job handlers — a job may run twice after a crash, and a
    transaction submitted twice is a real financial event, not a retry.
  • Visibility: queue depth, job age, and failure rate as metrics.

Acceptance criteria

  • Durable queue surviving restart
  • Handlers idempotent, verified by a test that runs a job twice
  • Dead-letter queue with an inspection path
  • Queue metrics exposed

Notes

Test the double-run case explicitly. "It only runs once" is an assumption that
holds until the first unclean shutdown.

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSThird CampaignCampaign: Third Campaignarea:coreServer bootstrap and pluginsdifficulty:hardDesign judgement required; subtle failure modespriority:highNeeded for the next milestonetype:featNew capability

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions