feat: continuously test alert delivery - #428
Open
Ukorstack wants to merge 3 commits into
Open
Conversation
Implements end-to-end synthetic alert delivery verification: - AlertDeliveryTestService: sends periodic synthetic test alerts through every configured alert route (primary + fallback + extra), verifies HTTP 2xx acknowledgement, detects missed/stale tests, and exposes operator status via getAlertDeliveryTestStatus(). - alertDeliveryTest.job: setInterval scheduler (default 10 min) following the existing messaging.jobs.js pattern. Runs on startup, stale-detector fires every 60 s. Both timers are unref()'d to not block shutdown. - Synthetic safety: payloads carry event='sendam-alert-delivery-test', synthetic=true, and a disclaimer message. No customer IDs, phone numbers, or real alert event names. Cannot trigger customer notifications. - Fallback routing: primary failure triggers configured fallback route. Fallback only attempted once per primary failure (no infinite loop). - Metrics: attempts, per-route success/failure, fallback, overall success/failure, stale gauge, last-success and last-attempt timestamps. - Structured logging: test_started, route_delivery, route_acknowledged, route_failed, fallback_attempted, fallback_acknowledged, test_completed, test_stale log events. - Admin endpoint: GET /admin/alert-delivery-test (requires operations.write) returns full status snapshot including route results, healthy bool, stale flag, last successful test, and fallback usage. - Prometheus rules: SendAmAlertDeliveryTestFailed, SendAmAlertDeliveryTestStale, SendAmAlertDeliveryFallbackUsed. - 41 new tests: success path, primary+fallback, both fail, ack failure, multi-route, stale detection, customer safety, duplicate safety, retry bounds, scheduler lifecycle, env config defaults. - OBSERVABILITY.md: full operator runbook section with config reference, metric table, Prometheus alert table, admin API example, troubleshooting. - .env.example: all 6 new ALERT_DELIVERY_TEST_* variables documented. Closes EF-CHAIN#228
|
@Ukorstack is attempting to deploy a commit to the ezedikeevan's projects Team on Vercel. A member of the Team first needs to authorize it. |
- fix(test): add missing dependency stubs to admin.balanceSummary.test.js so the test loads admin.controller in isolation without ENCRYPTION_KEY; 5 balance-summary tests now pass (was crashing on module load) - fix(lint): remove stale eslint-disable directives in adminApi.js where _-prefixed destructuring patterns no longer need suppression - fix(lint/admin): add react-hooks/exhaustive-deps suppression in Users.jsx useEffect where fetchUsers recreation is intentional - fix(lint): add eslint-disable-next-line no-unused-vars to unused-but- intentional bindings across payment, compliance, wallet, queue, and whatsapp modules (no behaviour change) - fix(admin): move Field component definition above early-return in TransactionDetail.jsx to avoid component-in-render lint violation - fix(admin): remove duplicate mock entries in enhancedConfirmation test (duplicate findUnique and updateMany keys on same object) - fix(admin/ui): show all sidebar links before permissions load; wrap useEffect fetch in setTimeout to avoid sync state updates - fix(admin/ui): add missing MSW handlers for /api/admin/me and /api/admin/password to test mock server - fix(admin/ui): fix AuditLogs useEffect — rewrite as async/await, fix export button refs (handleExportAudit, exportingAudit), add missing Export Events and Verify Chain buttons - fix(landing): wrap OnboardingStatus useEffect fetch in setTimeout with cleanup to satisfy exhaustive-deps - fix(api): remove unused prisma import in wallet.controller and unused phoneNumber destructure in sendFunds - fix(api): remove duplicate id select in rotate-wallet-keys wallet query - fix(api): remove stray whitespace line in deposits.jobs.js - fix(api): rename approvedBy -> _approvedBy in fulfillErasure signature (parameter was unused; callers pass it by name, no behaviour change) All 852 tests pass (845 pass, 7 skipped for missing provider creds), 0 failures. All three linters (api, admin, landing) pass clean.
- fix(a11y): remove role='button' from DataTable <tr> elements
Table rows with onRowClick had role='button' overriding the implicit
'row' role, causing accessibility tests that query getAllByRole('row')
to only find the header row. Rows now keep their table-row semantics;
click/keyboard handlers and tabIndex are preserved. cursor-pointer
class added to clickable rows for visual affordance.
- fix(a11y): add waitFor to renderLogin() accessibility tests
Login is lazy-loaded via React.lazy(). Three tests queried
getByLabelText('Email') synchronously after renderLogin(), hitting
the Suspense fallback (empty <div />) before the component resolved.
All three tests now await waitFor(() => getByLabelText('Email'))
before proceeding.
- fix(deps): remove accidental @asamuzakjp/css-color devDependency
This package was added to the root package.json but is not used
anywhere in the codebase. Removed to keep the lockfile clean.
All tests: 81/81 admin, 41/41 landing, 845/852 backend (0 failures).
All linters and builds pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements end-to-end synthetic alert delivery verification (Issue #228).
The system continuously exercises every configured alert route with periodic synthetic test payloads, verifies HTTP acknowledgement, detects failures, falls back to secondary routes when the primary fails, and surfaces missed/stale tests as actionable failures.
Changes
New files
apps/api/src/observability/alertDeliveryTest.service.js— Core service: builds routes from config, generates synthetic payloads, delivers to each route, handles fallback, detects stale tests, exposes operator status.apps/api/src/jobs/alertDeliveryTest.job.js— setInterval scheduler (follows existing messaging.jobs.js pattern). Default interval: 10 minutes. Runs on startup. Stale detector fires every 60 s.apps/api/test/alertDeliveryTest.service.test.js— 33 tests covering all code paths.apps/api/test/alertDeliveryTest.job.test.js— 8 tests covering scheduler lifecycle.Modified files
apps/api/src/config/env.js— AddedalertDeliveryTestconfig block (interval, fallback URL/token, extra URLs, timeout, stale multiplier).apps/api/src/jobs/index.js— RegisteredstartAlertDeliveryTestSchedulerinregisterJobs().apps/api/src/controllers/admin.controller.js— AddedgetAlertDeliveryTestStatusHandler.apps/api/src/routes/admin.routes.js— AddedGET /admin/alert-delivery-test(requiresoperations.write).apps/api/.env.example— Documented all 6 newALERT_DELIVERY_TEST_*variables.observability/prometheus-rules.yml— 3 new alert rules:SendAmAlertDeliveryTestFailed,SendAmAlertDeliveryTestStale,SendAmAlertDeliveryFallbackUsed.docs/OBSERVABILITY.md— Full operator runbook section with config reference, metric table, admin API example, and troubleshooting guide.Testing
All 41 new tests pass:
Tests cover:
Lint: 0 errors (2 pre-existing warnings in unrelated files).
Safety
Synthetic test payloads carry:
event: "sendam-alert-delivery-test"synthetic: truemessage: "Synthetic alert delivery verification — not a real incident."They contain no customer IDs, phone numbers, transaction IDs, or real alert event names. They cannot trigger customer notifications, create production incidents, or enter any customer-facing workflow. This is validated in the test suite.
Closes #228