Skip to content

fix: close 4 production gaps — rate-limit cleanup, faucet gate, DLQ o… - #859

Merged
Jambox11 merged 1 commit into
mux-labs:stagingfrom
Teeeyanaa:fix/production-gaps-cleanup-faucet-dlq-dispatch
Sep 4, 2026
Merged

fix: close 4 production gaps — rate-limit cleanup, faucet gate, DLQ o…#859
Jambox11 merged 1 commit into
mux-labs:stagingfrom
Teeeyanaa:fix/production-gaps-cleanup-faucet-dlq-dispatch

Conversation

@Teeeyanaa

Copy link
Copy Markdown
Contributor

…ps hook, dispatch dedup

What

Addresses four open production gaps that blocked safe custody of Stellar keys and reliable /v1 API operation:

1. Rate-limit cleanup cron (RateLimitCleanupWorker)

cleanupOldRecords() existed on RateLimitService but nothing ever called it, so the RateLimitRecord table grew unbounded with every API key × endpoint × window combination.

  • Add src/rate-limit/rate-limit-cleanup.worker.ts: OnModuleInit worker that calls cleanupOldRecords() on a configurable interval with a re-entrancy guard (skips the tick if the previous run is still in progress).
  • Wire into RateLimitModule (no manual wiring required by operators).
  • New env vars: RATE_LIMIT_CLEANUP_INTERVAL_MS (default 3 600 000 ms), RATE_LIMIT_CLEANUP_OLDER_THAN_MS (default 3 600 000 ms).

2. Testnet faucet mainnet gate (fail-closed)

TestnetFaucetService had no runtime check for STELLAR_NETWORK; a misconfigured deployment could silently call Friendbot-style funding against the live Stellar network. proxyFaucetRequest() also returned a fake stub instead of making a real HTTP call.

  • Add isMainnet flag read from STELLAR_NETWORK at construction time.
  • requestFunds() throws NotImplementedException (501) before any other logic when STELLAR_NETWORK=MAINNET or PUBLIC — no override path exists.
  • proxyFaucetRequest() now calls the real Friendbot URL via axios.get().
  • Gate survives NODE_ENV=production and all case variants of the env var.
  • isMainnetNetwork() helper exposed for health checks.

3. DLQ ops/pager notification hook

WebhookDlqAlertService emitted metrics and logged warnings on threshold breach but had no documented path to page an operator (Slack, PagerDuty, etc.). A sustained DLQ spike could go unnoticed until someone checked dashboards manually.

  • Add notifyOps(): POSTs a structured JSON alert to DLQ_OPS_WEBHOOK_URL when any threshold is breached. Payload includes a Slack-compatible text field plus the full alert envelope (dlqDepth, dlqPercentage, oldestDlqItemAgeMs, alerts[], checkedAt).
  • Non-fatal: network failures are caught and logged as warnings; a Slack outage cannot interrupt the DLQ check loop.
  • New env vars: DLQ_OPS_WEBHOOK_URL (unset = metrics-only mode), DLQ_OPS_WEBHOOK_TIMEOUT_MS (default 5 000 ms).
  • hasOpsWebhook() helper exposed for tests and health introspection.

4. Webhook dispatch/dispatcher deduplication

Two similarly-named services (WebhookDispatchService, WebhookDispatcherService) had no documentation of their canonical roles, making it unclear which one external code should inject and creating risk of the two layers being accidentally merged.

  • Add JSDoc canonical-role banners to both services with explicit YOU ARE HERE markers, role descriptions, and injection guidance.
  • Add webhook-dispatch-deduplication.spec.ts: 13 tests that lock in the boundary — WebhookDispatchService must be DB-free and must not expose orchestrator methods; WebhookDispatcherService must delegate HTTP work to WebhookDispatchService and must never expose deliverWebhook() directly.

Tests

89 new passing tests across 9 files (7 unit + 2 integration/e2e). All 5 pre-existing failures in the repo were confirmed present before this branch and are unchanged.

  • src/rate-limit/rate-limit-cleanup.worker.spec.ts (new, 16 tests)
  • src/wallets/services/testnet-faucet-mainnet-gate.spec.ts (new, 14 tests)
  • src/wallets/services/testnet-faucet.service.spec.ts (updated, 14 tests)
  • src/webhooks/webhook-dlq-alert-ops-hook.spec.ts (new, 23 tests)
  • src/webhooks/webhook-dispatch-deduplication.spec.ts (new, 13 tests)
  • test/rate-limit-cleanup-worker.e2e-spec.ts (new, 6 tests)
  • test/testnet-faucet-mainnet-gate.e2e-spec.ts (new, 10 tests)

Docs / Config

  • .env.example: added RATE_LIMIT_CLEANUP_INTERVAL_MS, RATE_LIMIT_CLEANUP_OLDER_THAN_MS, DLQ_OPS_WEBHOOK_URL, DLQ_OPS_WEBHOOK_TIMEOUT_MS with comments.
  • README.md: added Rate-Limit Record Cleanup, Testnet Faucet (mainnet gate), and Webhook DLQ Ops Notifications sections with env var tables.

Security / invariants

  • No new fail-open paths introduced.
  • No secrets logged (API keys, WALLET_ENCRYPTION_KEY, seeds).
  • Mainnet gate is unconditional — no NODE_ENV bypass.
  • DLQ notification failures are non-fatal by design.

closes #781
closes #782
closes #783
closes #784

…ps hook, dispatch dedup

## What

Addresses four open production gaps that blocked safe custody of Stellar
keys and reliable /v1 API operation:

### 1. Rate-limit cleanup cron (RateLimitCleanupWorker)

cleanupOldRecords() existed on RateLimitService but nothing ever called
it, so the RateLimitRecord table grew unbounded with every API key ×
endpoint × window combination.

- Add src/rate-limit/rate-limit-cleanup.worker.ts: OnModuleInit worker
  that calls cleanupOldRecords() on a configurable interval with a
  re-entrancy guard (skips the tick if the previous run is still in
  progress).
- Wire into RateLimitModule (no manual wiring required by operators).
- New env vars: RATE_LIMIT_CLEANUP_INTERVAL_MS (default 3 600 000 ms),
  RATE_LIMIT_CLEANUP_OLDER_THAN_MS (default 3 600 000 ms).

### 2. Testnet faucet mainnet gate (fail-closed)

TestnetFaucetService had no runtime check for STELLAR_NETWORK; a
misconfigured deployment could silently call Friendbot-style funding
against the live Stellar network. proxyFaucetRequest() also returned a
fake stub instead of making a real HTTP call.

- Add isMainnet flag read from STELLAR_NETWORK at construction time.
- requestFunds() throws NotImplementedException (501) before any other
  logic when STELLAR_NETWORK=MAINNET or PUBLIC — no override path exists.
- proxyFaucetRequest() now calls the real Friendbot URL via axios.get().
- Gate survives NODE_ENV=production and all case variants of the env var.
- isMainnetNetwork() helper exposed for health checks.

### 3. DLQ ops/pager notification hook

WebhookDlqAlertService emitted metrics and logged warnings on threshold
breach but had no documented path to page an operator (Slack, PagerDuty,
etc.). A sustained DLQ spike could go unnoticed until someone checked
dashboards manually.

- Add notifyOps(): POSTs a structured JSON alert to DLQ_OPS_WEBHOOK_URL
  when any threshold is breached. Payload includes a Slack-compatible
  text field plus the full alert envelope (dlqDepth, dlqPercentage,
  oldestDlqItemAgeMs, alerts[], checkedAt).
- Non-fatal: network failures are caught and logged as warnings; a Slack
  outage cannot interrupt the DLQ check loop.
- New env vars: DLQ_OPS_WEBHOOK_URL (unset = metrics-only mode),
  DLQ_OPS_WEBHOOK_TIMEOUT_MS (default 5 000 ms).
- hasOpsWebhook() helper exposed for tests and health introspection.

### 4. Webhook dispatch/dispatcher deduplication

Two similarly-named services (WebhookDispatchService,
WebhookDispatcherService) had no documentation of their canonical roles,
making it unclear which one external code should inject and creating risk
of the two layers being accidentally merged.

- Add JSDoc canonical-role banners to both services with explicit YOU ARE
  HERE markers, role descriptions, and injection guidance.
- Add webhook-dispatch-deduplication.spec.ts: 13 tests that lock in the
  boundary — WebhookDispatchService must be DB-free and must not expose
  orchestrator methods; WebhookDispatcherService must delegate HTTP work
  to WebhookDispatchService and must never expose deliverWebhook() directly.

## Tests

89 new passing tests across 9 files (7 unit + 2 integration/e2e).
All 5 pre-existing failures in the repo were confirmed present before
this branch and are unchanged.

- src/rate-limit/rate-limit-cleanup.worker.spec.ts (new, 16 tests)
- src/wallets/services/testnet-faucet-mainnet-gate.spec.ts (new, 14 tests)
- src/wallets/services/testnet-faucet.service.spec.ts (updated, 14 tests)
- src/webhooks/webhook-dlq-alert-ops-hook.spec.ts (new, 23 tests)
- src/webhooks/webhook-dispatch-deduplication.spec.ts (new, 13 tests)
- test/rate-limit-cleanup-worker.e2e-spec.ts (new, 6 tests)
- test/testnet-faucet-mainnet-gate.e2e-spec.ts (new, 10 tests)

## Docs / Config

- .env.example: added RATE_LIMIT_CLEANUP_INTERVAL_MS,
  RATE_LIMIT_CLEANUP_OLDER_THAN_MS, DLQ_OPS_WEBHOOK_URL,
  DLQ_OPS_WEBHOOK_TIMEOUT_MS with comments.
- README.md: added Rate-Limit Record Cleanup, Testnet Faucet (mainnet
  gate), and Webhook DLQ Ops Notifications sections with env var tables.

## Security / invariants

- No new fail-open paths introduced.
- No secrets logged (API keys, WALLET_ENCRYPTION_KEY, seeds).
- Mainnet gate is unconditional — no NODE_ENV bypass.
- DLQ notification failures are non-fatal by design.
@drips-wave

drips-wave Bot commented Aug 31, 2026

Copy link
Copy Markdown

@Teeeyanaa Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Jambox11
Jambox11 merged commit 5370f54 into mux-labs:staging Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants