feat(outbox): add durable transactional outbox + prioritized dispatcher - #332
Merged
robertocarlous merged 1 commit intoAug 19, 2026
Merged
Conversation
Every on-chain money movement (deposit, withdraw, agent rebalance, referral reward) was fire-and-forget: build a Stellar operation, submit it, handle failure locally, with no durable record of intent that survives a crash and no ordering/prioritization/fee-bumping under congestion. Adds src/outbox/ as the single choke point every write now passes through: a transactional outbox (intent + business row commit/roll back together), an idempotency-keyed atomic claim (no double-submission), priority-ordered dispatch (CRITICAL withdrawals never starve behind a NORMAL/LOW wave), retry/backoff/fee-bump, a compliance halt guard, per-signer serialization, admin tooling, and Prometheus metrics. A structural test fails CI the moment a money path bypasses the outbox. Closes Neurowealth#325
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Durable Outbox & Prioritized On-Chain Transaction Queue
Closes #325
Summary
Introduces
src/outbox/— a durable, transactional outbox and prioritized dispatcher that is now the single choke point every on-chain money movement passes through, replacing the fire-and-forget pattern where each caller independently built and submitted a Stellar operation.OutboxOpmodel. Every money-moving caller writes its intent (enqueueOutboxOp) inside the samedb.$transactionas the business row it derives (aTransaction, aReferralConversionleg) — "intent persisted" and "business state written" now commit or roll back together.kind:userId:businessRecordIdkey (src/outbox/idempotency.ts);enqueueOutboxOpis an upsert by that key.PENDING -> SUBMITTEDvia a single conditionalupdateMany— exactly one of any concurrent claimers wins, the rest no-op. Proven under a real Postgres intests/integration/outbox/dispatcher.integration.test.ts(excluded from the defaultnpm testrun, same convention asdeposit-withdraw.integration.test.ts— run manually per that file's header comment).tests/unit/outbox/stateMachine.test.ts.OUTBOX_MAX_ATTEMPTS; aSUBMITTEDop unconfirmed pastOUTBOX_SUBMITTED_TIMEOUT_MS(dispatcher crash or network congestion) is escalated to a fee-bumped resubmission (feeMultiplierthreaded throughsrc/stellar/contract.ts's write functions) up toOUTBOX_FEE_BUMP_MAX_ATTEMPTS, thenFAILEDwith the full attempt/error history retained.src/stellar/events.tsadditionally reconciles aSUBMITTEDop bytxHashon the same DB transaction that confirms theTransactionrow, closing out the crash-recovery case (dispatcher died between submit and observing its own confirmation).src/outbox/signerLock.ts— an account's ops never overlap (sequence-number safety), with configurable global/per-account in-flight caps.User.isActive(the same fieldsrc/middleware/authenticate.tsalready checks) before submitting — a user frozen mid-flight has queued ops halted, not just new requests rejected.tests/unit/outbox/structural.test.tsscans all ofsrc/and fails CI the moment any file outsidesrc/stellar/contract.ts/src/outbox/executors.tsimports a raw write function (depositForUser,withdrawForUser,triggerRebalance,payReferralReward).Caller migration
POST /deposit,POST /withdraw201withCONFIRMED/FAILED)executeDepositwith the HTTP path — gets durability for freedispatchInBackground, not awaited) — the loop moves to the next batch immediately, per the issue's explicit non-blocking requirementDesign note on scope: deposit/withdraw/referral keep their existing synchronous response contract (still return
CONFIRMED/FAILEDimmediately) rather than moving to an async202 PENDINGenvelope. This was a deliberate choice to keep the blast radius bounded — a full async conversion would ripple into WhatsApp/Telegram reply formatting, recurring-deposit next-run logic, and referral leg bookkeeping, none of which this issue asked to change. The durability win (transactional write-before-submit, retriable via the background sweep or admin force-retry) is real regardless; a follow-up could make the HTTP layer itself async on top of this.Admin & observability
GET /api/admin/outbox(list/filter),GET /api/admin/outbox/stats,GET /api/admin/outbox/:id,POST /api/admin/outbox/:id/retry(FAILED → PENDING),POST /api/admin/outbox/:id/cancel(PENDING-only) — newoutbox:read/outbox:writeadmin scopes, fullyAdminAuditLog-audited.outbox_ops_total{kind,priority,outcome},outbox_queue_depth{status,priority},outbox_op_latency_seconds{kind},outbox_fee_bump_total{kind},outbox_stuck_submitted.docs/OUTBOX.md: state machine diagram, idempotency/retry/fee-bump policy, ordering guarantees, admin API, config reference.Migration
prisma/migrations/20260819120000_add_outbox_op/adds theOutboxOptable + 4 enums, with arollback.sql(verified to drop cleanly against a fresh DB). No existing table is altered.Testing
tests/unit/outbox/{stateMachine,idempotency,structural}.test.ts— pure state machine, priority/starvation, backoff bounds, idempotency-key derivation, no-bypass guarantee.tests/integration/outbox/dispatcher.integration.test.ts(real DB, manual-run convention) — atomic claim race, kill-the-worker recovery (crashedSUBMITTEDop is recovered exactly once, not lost or double-executed), frozen-user halt, priority ordering under the real claim path.tests/integration/agent/rebalance.integration.test.ts,tests/unit/referral/service.test.ts,tests/integration/deposit-withdraw.integration.test.ts(manual-run) for the new outbox-mediated call paths — all existing assertions preserved.npm test— 73 suites / 1109 tests passing.npm run lint,npm run format:check,npm run build,npm run validate:specall clean.Out of scope (per the issue)
Multi-signer/HSM signing, cross-chain submission, replacing the event listener as confirmation source of truth, and running more than one dispatcher process (the atomic claim would make it safe; the in-process signer mutex would not coordinate across processes without an additional distributed lock — noted in
docs/OUTBOX.md).