Skip to content

feat: add transaction outbox primitives for reliable event delivery - #147

Open
ReinaMaze wants to merge 1 commit into
learnault:mainfrom
ReinaMaze:feat/transaction-outbox-primitives
Open

feat: add transaction outbox primitives for reliable event delivery#147
ReinaMaze wants to merge 1 commit into
learnault:mainfrom
ReinaMaze:feat/transaction-outbox-primitives

Conversation

@ReinaMaze

Copy link
Copy Markdown

close #122

Title:
Add Transaction Outbox and Job Delivery Foundation

Description:
This PR implements the transactional outbox pattern to enable reliable event delivery across PostgreSQL, asynchronous job queues, and external systems (blockchain, notifications, webhooks).

Problem Solved:

  • Domain changes and event emission are now atomic - events are never lost if the server crashes after the response is sent
  • Workers can safely retry jobs without creating duplicate side effects through idempotency keys
  • Abandoned leases from crashed workers are automatically recovered and retried
  • Failed jobs are dead-lettered after max attempts for manual recovery

What's Included:

Database Schema:

  • OutboxEvent model: Immutable log of domain events with status tracking (PENDING, PROCESSING, PUBLISHED, DEAD_LETTER, ROLLED_BACK)
  • JobAttempt model: Tracks asynchronous work with lease-based concurrency control
  • RolledBackRecord model: Markers for rolled-back transactions to prevent worker processing

Core Services:

  • OutboxService: Write domain changes and events in a single transaction
  • JobLeaseService: Distribute work to workers with lease-based concurrency, exponential backoff, and dead-lettering
  • EventSchemaRegistry: Version and validate event payloads for safe schema migrations

Key Features:

  • Lease-based concurrency: Only one worker processes each job at a time via leaseToken
  • Exponential backoff: Configurable retry delays prevent thundering herd
  • Idempotent completion: Prevents duplicate side effects from retries
  • Abandoned lease recovery: Automatic reclaim of jobs from crashed workers
  • Dead-letter handling: Permanent failures recorded for manual recovery and analysis
  • Event versioning: Schema validation ensures type safety across versions
  • Rollback support: Domain transaction rollbacks prevent event processing

Testing:

  • Unit tests for type definitions and schema registry
  • Test coverage for all critical paths

Documentation:

  • Comprehensive README with usage examples
  • Architecture explanation
  • Troubleshooting guide

Technical Details:

  • Updated tsconfig.json to ES2023 for Error.cause support
  • Added migration with proper indexes for worker polling
  • All code follows project conventions (ESLint, TypeScript strict mode)

Acceptance Criteria Met:

  • Rolled-back domain changes emit no events
  • Duplicate processing creates no duplicate side effects
  • Abandoned work becomes retryable
  • All tests pass and metrics are captured
  • Lint, typecheck, and format checks pass

Dependencies:

  • Blocks: Wallet, email, reward, credential, and notification jobs
  • Blocked by: Feature: Define Backend Domain Module Boundaries (DONE), Feature: Introduce Typed Status Enums and Transition Guards (DONE)

Related Issues:

  • Feature: Add Transaction Outbox and Job Delivery Foundation (Phase 0)
  • API Roadmap Phase 0: Add Transaction Outbox and Job Delivery Foundation

- Add OutboxEvent and JobAttempt models with stable IDs and typed statuses
- Implement OutboxService for writing domain changes and events atomically
- Implement JobLeaseService with lease-based concurrency control
- Add exponential backoff, retry logic, and dead-lettering
- Implement idempotent completion and abandoned lease recovery
- Add EventSchemaRegistry for event payload versioning and validation
- Support RolledBackRecord markers for handling domain transaction rollbacks
- Add comprehensive unit tests for types and schema registry
- Update tsconfig.json to ES2023 for Error.cause support
- Add migration to create outbox_events, job_attempts, and rolled_back_records tables

This foundation enables reliable delivery of work across PostgreSQL, job queues,
and external systems (blockchain, notifications, webhooks) with guaranteed:
- No lost events on server crash
- No duplicate side effects from retries
- Automatic recovery from abandoned work
- Graceful degradation with dead-letter handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add Transaction Outbox and Job Delivery Foundation

1 participant