Durable Outbox & Prioritized On-Chain Transaction Queue

[robertocarlous]

Problem Statement

Every money movement in this platform is fire-and-forget from the caller's perspective: the deposit route, the agent loop's rebalances, recurring deposits, and referral payouts each independently construct a Stellar operation, submit it, and handle failure locally. There is no shared, durable record of intent that survives a crash, a restart, or a mid-flight RPC timeout; there is no global ordering or prioritization; and when the network is congested there is no fee-bumping strategy. A process that dies between "transactionally persisted the intent" and "confirmed on-chain" relies on the event listener's reconciliation to guess what happened. This issue introduces a durable outbox + prioritized dispatcher for all on-chain operations — the single choke point that makes every money move atomic, retriable, ordered, and observable.

Current State

• Money moves go through src/stellar/contract.ts's executeWriteContractCall and src/stellar/wallet.ts (agent-signed), called from: deposit (src/routes/deposit.ts), withdraw (src/routes/withdraw.ts), the agent loop (src/agent/loop.ts), recurring deposits (src/jobs/recurringDeposits.ts), referral payouts (src/jobs/referralPayout.ts), and (for fiat settlement) reconciliation. Each has its own retry/error shape; DeadLetterEvent is the closest thing to a shared failure record and is only fed by the event listener.
• ProcessedEvent + EventCursor give the inbound side exactly-once-ish semantics; there is no equivalent for outbound intents. docs/REFERRAL_PROGRAM.md shows the pain explicitly: payouts must be "independently retriable" and a stuck row "visible/queryable".

Proposed Solution

1. Durable intent (outbox) model

model OutboxOp {
  id            String        @id @default(uuid())
  // idempotency anchor — every existing money path already has one; unify them here
  idempotencyKey String       @unique
  userId        String
  kind          OutboxOpKind  // DEPOSIT | WITHDRAW | REBALANCE | RECURRING_DEPOSIT | REFERRAL_REWARD | YIELD_CLAIM
  actor         String        // USER | AGENT | SYSTEM
  payload       Json          // the exact, validated operation: { method, params, asset, amount, destination }
  priority      OutboxPriority // CRITICAL (user withdrawals) | NORMAL (recurring/referrals) | LOW (agent rebalances)
  status        OutboxStatus  // PENDING | SUBMITTED | CONFIRMED | FAILED | CANCELLED
  txHash        String?
  attempts      Int           @default(0)
  nextAttemptAt DateTime?
  error         String?
  submittedAt   DateTime?
  confirmedAt   DateTime?
  createdAt     DateTime      @default(now())
  updatedAt     DateTime      @updatedAt
}

• Transactional outbox: each money-path service writes its intent inside the same DB transaction as whatever business state it derives from (e.g. the withdrawal record, the referral conversion, the recurring-deposit run). "Intent persisted" and "business state written" commit or roll back together — this is the atomicity upgrade the current callers lack.
• Idempotency anchor: idempotencyKey is derived deterministically (e.g. kind:user:businessRecordId), and the dispatcher guarantees an op is submitted at most once even after crashes/restarts/retries. Existing callers keep their own anchors; the outbox centralizes them.
• Write path only: the outbox is an append + state-machine. No code path may construct a Stellar operation that bypasses it (structural test over the call graph, mirroring the existing import-graph tests).

2. Dispatcher (src/outbox/dispatcher.ts)

• A worker (scheduled job registered in src/index.ts, or a self-scheduling loop like src/agent/loop.ts) that:
  • Picks PENDING ops ordered by priority then createdAt (a NORMAL wave must not starve a CRITICAL withdrawal during a rebalance storm — test this).
  • Submits via executeWriteContractCall, transitions to SUBMITTED with the tx hash, and lets the existing event listener (src/stellar/events.ts) be the confirmation oracle: a CONFIRMED ProcessedEvent matching the hash transitions the op to CONFIRMED.
  • Retries with exponential backoff + jitter and a fee-bumping strategy: when submission fails due to congestion (or a SUBMITTED op hasn't confirmed within a window), a higher-fee resubmission is attempted up to a documented cap — mirrored to the Stellar client's existing transaction-expiration handling.
  • Moves permanently-failed ops to FAILED with the error and emits a deduplicated operational alert (via alertingService) plus a user-facing webhook event.
• Concurrency control: a per-account and global in-flight cap (documented) so a burst can't saturate the network or the nonce space; a per-kind ordering guarantee within an account where required (e.g. withdrawals on the same wallet must sequence; the agent-signed wallet's nonce semantics dictate the constraint — document and enforce).

3. Caller migration

• Deposit/withdraw (src/routes/deposit.ts / withdraw.ts): create the outbox op transactionally, return the op id + PENDING (the caller already models Transaction.status=PENDING, so the response semantics are compatible). Confirmation still flows from the event listener → Transaction → outbox.
• Agent loop (src/agent/loop.ts): a rebalance that would move funds creates a LOW op; the loop must not wait on it (non-blocking), preserving the current loop cadence while gaining durability.
• Recurring deposits (src/jobs/recurringDeposits.ts) and referral payouts (src/jobs/referralPayout.ts): replace their bespoke retry loops with outbox ops (their lastRunStatus/payoutError fields become mirrors of outbox state — keep them in sync transactionally, and document the invariant).
• Settlement/cleanup: the fiat reconciliation path's settlement step already keys off the event listener; it should read outbox-confirmed state where available rather than re-deriving.

4. Admin & observability

• Admin endpoints (scoped keys, AdminAuditLog-audited): list/query outbox ops, force-retry a FAILED op, cancel a PENDING op (only those not yet submitted), and inspect per-kind/priority throughput.
• Prometheus metrics: op latency by stage (persisted→submitted→confirmed), retry counts, backoff/fee-bump events, queue depth by priority, and a gauge for ops SUBMITTED but unconfirmed longer than a window (the "lost in flight" alarm).
• docs/ writeup of the outbox state machine, retry/fee-bump policy, and ordering guarantees.

Edge Cases & Failure Modes

• Crash after DB commit, before submission: on restart the dispatcher re-submits — idempotency key + listener dedup make this safe (assert with a kill-the-worker integration test).
• Submitted but unconfirmed for a long time: timeout escalates to fee-bump; past a hard cap → FAILED with full audit trail (the Transaction row may still confirm later — reconcile via txHash, documented).
• Double-submission race: two dispatchers pick the same op — claim via an atomic PENDING→SUBMITTED conditional update; the loser is a no-op (test).
• Nonce conflicts on the shared agent wallet: the dispatcher must serialize ops sharing a signer; document the per-signer mutex and test ordering.
• Op references a delisted protocol / invalid params: validate payload against the same schema the route would use, at persist time (fail fast at enqueue), not at submit time.
• User frozen/blocked mid-flight (compliance): the dispatcher must consult the same central guard as the routes so a freeze also halts already-queued user ops (not just new ones).

Security & Privacy Considerations

• The outbox payload is the validated operation — same allowlist discipline as request validation; never store keys; the signer only ever reads the ops it's dispatched to.
• Admin force-retry/cancel is a powerful operation: scoped keys only, fully audited, and cancel limited to unsent ops.
• No PII in outbox rows beyond userId (payload carries on-chain addresses/amounts, which the account already owns).
• The structural "no bypass" test is the load-bearing security property — a future money path that skips the outbox must fail CI.

Out of Scope

• Multi-signer/HSM signing infrastructure (the dispatcher stays on the existing agent-signed path).
• Cross-chain submission (Stellar-only; the abstraction should make adding chains clean, but the executor stays Stellar).
• Replacing the event listener (confirmation oracle stays where it is).

Suggested Implementation Plan

1. Schema + migration (OutboxOp + enums) + idempotency-key derivation helper.
2. Pure state-machine + priority-ordering module (unit tests: transitions, ordering, starvation, race claims).
3. Dispatcher worker with retry/backoff/fee-bump and per-signer serialization.
4. Migrate the four money paths to transactional outbox writes; delete their bespoke retry loops.
5. Admin endpoints, metrics, docs, kill-the-worker + double-submission integration tests.

Acceptance Criteria

• All on-chain money movements write a transactional outbox intent; a structural test fails on any bypass
• Dispatcher claims ops atomically (no double-submission), respects priority ordering without starvation, and serializes per signer
• Confirmation flows from the event listener (the existing oracle); fee-bump on congestion up to a documented cap; hard cap → FAILED with audit trail
• Caller migration: deposit/withdraw/recurring/referral all enqueue transactionally; agent rebalances are non-blocking LOW ops
• Kill-the-worker restart test: no op is lost or double-executed; a submitted-but-unconfirmed op escalates then reconciles
• Freeze/compliance guard consulted at dispatch time — queued ops of a frozen user halt
• Admin force-retry/cancel (unsent only) + Prometheus metrics for latency/retries/queue/fee-bumps
• docs/openapi.yaml updated; unit + integration tests green

[Jenix777]

Hello Maintainer

I'd like to take on issue #325. I understand the core problem: money movement (deposits, agent rebalances, recurring deposits, referral payouts) is currently fire-and-forget — each path builds and submits its own Stellar operation with local failure handling, so there's no durable record of intent that survives a crash or mid-flight RPC timeout, no global ordering/prioritization across operations, and no fee-bumping strategy under network congestion. Right now recovery depends on the event listener reconciling after the fact rather than the system knowing what it intended to do.

My proposed approach: introduce a single outbox table that all four call sites write to before submission (persist intent, then attempt), replacing ad hoc submission logic with a shared dispatcher that pulls from a priority-ordered queue, tracks state through to on-chain confirmation, and handles retries with fee-bumping when transactions stall. This makes the outbox the single choke point you mentioned, rather than four independent submission paths.

Could you assign this to me? I'd like to align on schema/priority-tier design with you before implementing, given how central this is to the platform.

Thanks
Jennickson

[xeladev4]

Hi maintainer, I’d like to work on this issue.
I have experience building Stellar projects. I focus on clean implementations.
I’ll ensure the solution follows the project’s architecture and security best practices.

[grantfox-oss]

🦊 GrantFox — @xeladev4 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #325)
2. Your PR will be reviewed by the Neurowealth maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Third Campaign campaign!

@xeladev4's PR #332 was approved and merged by @robertocarlous.

🏆 @xeladev4: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (955 total points). Track your full progress on GrantFox.

👏 Great work, @xeladev4! Keep contributing to Neurowealth.