Skip to content

feat(outbox): add durable transactional outbox + prioritized dispatcher - #332

Merged
robertocarlous merged 1 commit into
Neurowealth:mainfrom
xeladev4:feat/issue-325-outbox-dispatcher
Aug 19, 2026
Merged

feat(outbox): add durable transactional outbox + prioritized dispatcher#332
robertocarlous merged 1 commit into
Neurowealth:mainfrom
xeladev4:feat/issue-325-outbox-dispatcher

Conversation

@xeladev4

Copy link
Copy Markdown
Contributor

Durable Outbox & Prioritized On-Chain Transaction Queue

Closes #325

Summary

Introduces src/outbox/ — a durable, transactional outbox and prioritized dispatcher that is now the single choke point every on-chain money movement passes through, replacing the fire-and-forget pattern where each caller independently built and submitted a Stellar operation.

  • Durable intent: new OutboxOp model. Every money-moving caller writes its intent (enqueueOutboxOp) inside the same db.$transaction as the business row it derives (a Transaction, a ReferralConversion leg) — "intent persisted" and "business state written" now commit or roll back together.
  • Idempotency: deterministic kind:userId:businessRecordId key (src/outbox/idempotency.ts); enqueueOutboxOp is an upsert by that key.
  • Atomic claim: PENDING -> SUBMITTED via a single conditional updateMany — exactly one of any concurrent claimers wins, the rest no-op. Proven under a real Postgres in tests/integration/outbox/dispatcher.integration.test.ts (excluded from the default npm test run, same convention as deposit-withdraw.integration.test.ts — run manually per that file's header comment).
  • Priority ordering: CRITICAL (user withdrawals) → NORMAL (deposits, recurring deposits, referral rewards) → LOW (agent rebalances), FIFO within a tier. A CRITICAL withdrawal never waits behind an arbitrarily large NORMAL/LOW wave — unit-tested in tests/unit/outbox/stateMachine.test.ts.
  • Retry / backoff / fee-bump: transient failures get full-jitter exponential backoff up to OUTBOX_MAX_ATTEMPTS; a SUBMITTED op unconfirmed past OUTBOX_SUBMITTED_TIMEOUT_MS (dispatcher crash or network congestion) is escalated to a fee-bumped resubmission (feeMultiplier threaded through src/stellar/contract.ts's write functions) up to OUTBOX_FEE_BUMP_MAX_ATTEMPTS, then FAILED with the full attempt/error history retained.
  • Confirmation oracle: the dispatcher's own submission already awaits on-chain confirmation; src/stellar/events.ts additionally reconciles a SUBMITTED op by txHash on the same DB transaction that confirms the Transaction row, closing out the crash-recovery case (dispatcher died between submit and observing its own confirmation).
  • Per-signer serialization + concurrency caps: src/outbox/signerLock.ts — an account's ops never overlap (sequence-number safety), with configurable global/per-account in-flight caps.
  • Compliance halt guard: dispatch consults User.isActive (the same field src/middleware/authenticate.ts already checks) before submitting — a user frozen mid-flight has queued ops halted, not just new requests rejected.
  • No-bypass structural guarantee: tests/unit/outbox/structural.test.ts scans all of src/ and fails CI the moment any file outside src/stellar/contract.ts / src/outbox/executors.ts imports a raw write function (depositForUser, withdrawForUser, triggerRebalance, payReferralReward).

Caller migration

Path Kind Priority Dispatch
POST /deposit, POST /withdraw DEPOSIT / WITHDRAW NORMAL / CRITICAL Enqueued transactionally, dispatched inline and awaited — HTTP response contract unchanged (still synchronous 201 with CONFIRMED/FAILED)
Recurring deposits DEPOSIT NORMAL Shares executeDeposit with the HTTP path — gets durability for free
Referral payouts REFERRAL_REWARD NORMAL Enqueued transactionally, dispatched inline and awaited
Agent rebalance REBALANCE LOW Enqueued transactionally, then fire-and-forget (dispatchInBackground, not awaited) — the loop moves to the next batch immediately, per the issue's explicit non-blocking requirement

Design note on scope: deposit/withdraw/referral keep their existing synchronous response contract (still return CONFIRMED/FAILED immediately) rather than moving to an async 202 PENDING envelope. This was a deliberate choice to keep the blast radius bounded — a full async conversion would ripple into WhatsApp/Telegram reply formatting, recurring-deposit next-run logic, and referral leg bookkeeping, none of which this issue asked to change. The durability win (transactional write-before-submit, retriable via the background sweep or admin force-retry) is real regardless; a follow-up could make the HTTP layer itself async on top of this.

Admin & observability

  • GET /api/admin/outbox (list/filter), GET /api/admin/outbox/stats, GET /api/admin/outbox/:id, POST /api/admin/outbox/:id/retry (FAILED → PENDING), POST /api/admin/outbox/:id/cancel (PENDING-only) — new outbox:read/outbox:write admin scopes, fully AdminAuditLog-audited.
  • Prometheus: outbox_ops_total{kind,priority,outcome}, outbox_queue_depth{status,priority}, outbox_op_latency_seconds{kind}, outbox_fee_bump_total{kind}, outbox_stuck_submitted.
  • docs/OUTBOX.md: state machine diagram, idempotency/retry/fee-bump policy, ordering guarantees, admin API, config reference.

Migration

prisma/migrations/20260819120000_add_outbox_op/ adds the OutboxOp table + 4 enums, with a rollback.sql (verified to drop cleanly against a fresh DB). No existing table is altered.

Testing

  • tests/unit/outbox/{stateMachine,idempotency,structural}.test.ts — pure state machine, priority/starvation, backoff bounds, idempotency-key derivation, no-bypass guarantee.
  • tests/integration/outbox/dispatcher.integration.test.ts (real DB, manual-run convention) — atomic claim race, kill-the-worker recovery (crashed SUBMITTED op is recovered exactly once, not lost or double-executed), frozen-user halt, priority ordering under the real claim path.
  • Updated tests/integration/agent/rebalance.integration.test.ts, tests/unit/referral/service.test.ts, tests/integration/deposit-withdraw.integration.test.ts (manual-run) for the new outbox-mediated call paths — all existing assertions preserved.
  • Full suite: npm test — 73 suites / 1109 tests passing. npm run lint, npm run format:check, npm run build, npm run validate:spec all clean.

Out of scope (per the issue)

Multi-signer/HSM signing, cross-chain submission, replacing the event listener as confirmation source of truth, and running more than one dispatcher process (the atomic claim would make it safe; the in-process signer mutex would not coordinate across processes without an additional distributed lock — noted in docs/OUTBOX.md).

Every on-chain money movement (deposit, withdraw, agent rebalance, referral
reward) was fire-and-forget: build a Stellar operation, submit it, handle
failure locally, with no durable record of intent that survives a crash and
no ordering/prioritization/fee-bumping under congestion.

Adds src/outbox/ as the single choke point every write now passes through:
a transactional outbox (intent + business row commit/roll back together),
an idempotency-keyed atomic claim (no double-submission), priority-ordered
dispatch (CRITICAL withdrawals never starve behind a NORMAL/LOW wave),
retry/backoff/fee-bump, a compliance halt guard, per-signer serialization,
admin tooling, and Prometheus metrics. A structural test fails CI the
moment a money path bypasses the outbox.

Closes Neurowealth#325
@robertocarlous
robertocarlous merged commit 0372457 into Neurowealth:main Aug 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Durable Outbox & Prioritized On-Chain Transaction Queue

2 participants