Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 14 additions & 128 deletions docs/ALERTS.md
Original file line number Diff line number Diff line change
@@ -1,134 +1,20 @@
# Custom Price & Yield Alert Rules
# Alert Rules, Acknowledgement, Snooze & Escalation (#366)

User-defined rules that proactively notify a user when a market or portfolio
condition they care about is met — e.g. "tell me if Blend's APY drops below 5%"
or "alert me if my portfolio value falls under $1,000".
NeuroWealth provides user-defined alert rules watching portfolio metrics (`PROTOCOL_APY`, `PORTFOLIO_VALUE`, `POSITION_DRAWDOWN`, `DRIFT`, `VOLATILITY_REGIME`, `ANOMALY`).

This is **end-user** alerting and is deliberately distinct from the operator-
facing Prometheus/Grafana alerting in [`OBSERVABILITY.md`](./OBSERVABILITY.md)
(`agent_loop_status`, `cursor_lag_ledgers`, `dlq_size`, …), which watches system
health for on-call engineers rather than portfolio conditions for users.
## Concepts & Control Flow

- Data model: `AlertRule` in [`prisma/schema.prisma`](../prisma/schema.prisma)
- Evaluation core (pure, unit-tested): [`src/services/alertEvaluator.ts`](../src/services/alertEvaluator.ts)
- Scheduled job: [`src/jobs/alertRules.ts`](../src/jobs/alertRules.ts)
- CRUD API: [`src/routes/alerts.ts`](../src/routes/alerts.ts)
- Delivery: reuses [`src/services/webhookDispatcher.ts`](../src/services/webhookDispatcher.ts)
(webhook) and [`src/whatsapp/formatters.ts`](../src/whatsapp/formatters.ts) (WhatsApp)
- **Cooldown (`cooldownMinutes`)**: Restricts re-firing frequency while a condition remains true across consecutive ticks.
- **Snooze (`POST /api/v1/alerts/:id/snooze`)**: Temporarily mutes an alert rule for a duration (clamped to max 30 days). Evaluation is skipped while snoozed, but would-be fires are recorded in `AlertFire` history with `suppressedBySnooze: true` for auditing. Auto-expires emitting `alert.snooze_expired`.
- **Acknowledge (`POST /api/v1/alerts/:id/ack`)**: Marks an alert episode as "seen" by linking an `AlertAck` to `AlertFire` records. Resets the escalation counter. Can be invoked via API or via signed single-use `ackToken` in WhatsApp/Telegram/Email notifications.
- **Escalation**: Triggers when the count of **un-acknowledged** fires (`AlertFire` where `ackId IS NULL`) within a rule's window reaches `escalationThreshold`. Escalated fires set `escalated = true` and deliver to `escalationChannel`.

## Rule model
---

A rule is a single condition (compound/multi-condition rules are out of scope
for v1):
## API Endpoints

| Field | Meaning |
| ----------------- | ------------------------------------------------------------------- |
| `metric` | `PROTOCOL_APY`, `PORTFOLIO_VALUE`, or `POSITION_DRAWDOWN` |
| `protocolName` | required for `PROTOCOL_APY`, rejected for the other metrics |
| `comparator` | `LT`, `LTE`, `GT`, `GTE` |
| `threshold` | compared against the observed value (units below) |
| `deliveryChannel` | `WEBHOOK`, `WHATSAPP`, or `BOTH` |
| `cooldownMinutes` | minimum gap between notifications for this rule (default 60) |
| `lastFiredAt` | when the rule last fired; drives cooldown |
| `isActive` | inactive rules are never evaluated |

### Units per metric

- **`PROTOCOL_APY`** — threshold and observed value are **percentages**
(`5` == 5%). `ProtocolRate.supplyApy` is stored as a fraction (`0.05`), so the
evaluator scales it by 100 before comparing.
- **`PORTFOLIO_VALUE`** — threshold and observed value are the **USD sum of the
user's ACTIVE positions' `currentValue`**.
- **`POSITION_DRAWDOWN`** — threshold and observed value are a **percentage
decline from a reference peak** (see below).

## POSITION_DRAWDOWN reference window

"Drawdown" is meaningless without a reference point, so we fix one explicitly:

> Drawdown is measured against the **rolling 30-day peak of the user's total
> portfolio value**.

The peak is the maximum of:

1. every historical whole-portfolio value reconstructed from `YieldSnapshot`
rows (`principalAmount + yieldAmount`, summed across the user's positions per
snapshot instant) within the trailing 30 days, and
2. the current total portfolio value.

Including the current value as a candidate means a fresh all-time high reports
**0% drawdown** rather than a spurious decline against a stale sample.

```
drawdown% = max(0, (peak - current) / peak * 100)
```

The window length is `WINDOW_DAYS` in [`src/jobs/alertRules.ts`](../src/jobs/alertRules.ts).

## Evaluation & cooldown

The job runs on a fixed interval (`ALERT_RULES_INTERVAL_MS`, default 60s). On
each tick it loads all `isActive` rules and, for each:

1. Observes the current value for the rule's metric.
2. Checks the comparator against the threshold.
3. If the condition holds, **atomically claims a fire** with an `updateMany`
guarded on `{ id, isActive, lastFiredAt outside cooldown }`, setting
`lastFiredAt = now`. Only if that update matches exactly one row does it
deliver.

The cooldown is essential: a rule sitting right at its threshold would otherwise
fire on every tick. With it, a rule notifies **at most once per
`cooldownMinutes`**. The condition does **not** have to flip false→true again —
if it is still true once the cooldown elapses, the rule re-fires.

### Edge cases

- **Condition true across many ticks** — cooldown suppresses repeats; the rule
stays active and re-fires after the cooldown if still true.
- **Rule deleted/deactivated mid-tick** — the atomic fire-claim matches 0 rows,
so delivery is skipped silently (no error, no send to a gone rule).
- **Protocol delisted** — a `PROTOCOL_APY` rule whose protocol has no
`ProtocolRate` row is **auto-deactivated** (`isActive = false`) with a logged
reason, rather than evaluated against missing/stale data.

## Delivery & failed-delivery retry policy

Delivery reuses the existing HMAC-signed webhook dispatcher
(`dispatchWebhookEvent('alert_rule.triggered', …)`) and/or the Twilio WhatsApp
sender. No new unsigned delivery path is introduced.

**Decision (per issue #289): alert deliveries reuse `dispatchWebhookEvent`
as-is and get no additional retry sweep beyond its synchronous 3-attempt
exponential backoff (1s/2s/4s).**

Rationale: alerts are about a *live* condition. A separate sweep that later
replays a `FAILED` delivery could fire a stale alert for a condition that has
since reversed. Instead:

- If **all** requested channels hard-fail during a fire, the job **rolls back
`lastFiredAt`** to its prior value, so the next tick re-evaluates the *current*
condition and retries if it still holds (bounded by cooldown). A transient
failure therefore self-heals on the following tick without replaying stale
data.
- The webhook dispatcher still persists a `WebhookDelivery` row with
`status = FAILED` for observability, exactly as for every other event.

If durable, at-least-once alert delivery is required later, the follow-up is a
dedicated retry sweep over `FAILED` `WebhookDelivery` rows — explicitly out of
scope here.

## Configuration

| Env var | Default | Meaning |
| ------------------------ | ------- | ------------------------------------ |
| `ALERT_RULES_INTERVAL_MS`| `60000` | Evaluation tick interval (ms) |

## Conversational management (WhatsApp)

Alert rules can be managed over WhatsApp via the NLP intents in
[`src/nlp/parser.ts`](../src/nlp/parser.ts) (`alert_create`, `alert_list`,
`alert_delete`), handled in [`src/whatsapp/handler.ts`](../src/whatsapp/handler.ts).
As with the other intents (#281/#282), the intent union, the `KNOWN_ACTIONS`
allowlist, and the handler switch are kept in sync **manually**. WhatsApp-created
rules default to `WHATSAPP` delivery.
| Method | Endpoint | Description |
| :--- | :--- | :--- |
| `POST` | `/api/v1/alerts/:id/snooze` | Mute rule for `durationMinutes` or `until` date |
| `POST` | `/api/v1/alerts/:id/ack` | Acknowledge alert fire(s) using `fireId` or `ackToken` |
| `GET` | `/api/v1/alerts/:id/fires` | View paginated fire history with ack & snooze status |
22 changes: 22 additions & 0 deletions docs/NOTIFICATIONS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Email Delivery Channel & Notifications (#367)

NeuroWealth supports `EMAIL` as a first-class delivery channel for alert rules, digests, and security notices alongside `WEBHOOK` and `WHATSAPP`.

## Verified Opt-In Requirement

To protect deliverability and prevent spam, email addresses must pass a double opt-in verification flow before receiving notifications:

1. **Request Verification**: `POST /api/v1/notifications/email` with `{ "email": "user@example.com" }`.
- Sends a verification email with a signed 24h single-use token.
- Address is set to `status: "PENDING"`.
2. **Confirm Address**: User clicks link `GET /api/v1/notifications/email/verify?token=...`.
- Address is set to `status: "VERIFIED"`.
- Email delivery channel can now be selected on alert rules.

---

## Mailer Architecture & Bounce Handling

- **Provider Abstraction (`MailProvider`)**: Supports AWS `SES` and `SMTP` (Nodemailer), selected via `MAIL_PROVIDER` environment variable. Uses a health ledger for automatic failover.
- **Mandatory Plaintext**: All templates generate both HTML and plaintext parts with unsubscribe / manage-preferences links.
- **Provider Webhooks (`POST /api/v1/webhooks/mail`)**: Signature-verified callback endpoint processing bounces and spam complaints. Hard bounces or complaints update status to `BOUNCED` / `COMPLAINED` / `SUPPRESSED` and emit `notification.email_suppressed`.
60 changes: 60 additions & 0 deletions docs/REALTIME_STREAMING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Real-Time Streaming (WebSocket & Server-Sent Events) (#369)

NeuroWealth provides two real-time event streaming transports backed by a shared event hub, topic authorization model, and durable per-user sequence stream (`UserEvent`):

1. **WebSocket (`/api/v1/ws`)**: Full-duplex transport suitable for interactive applications.
2. **Server-Sent Events (`/api/v1/stream/sse`)**: Plain HTTP GET fallback transport ideal for environments where WebSockets are blocked (corporate proxies, edge runtimes, curl/script consumers).

---

## Transport Comparison

| Feature | WebSocket (`/api/v1/ws`) | Server-Sent Events (`/api/v1/stream/sse`) |
| :--- | :--- | :--- |
| **Protocol** | `ws://` / `wss://` | Standard HTTP `GET` (`text/event-stream`) |
| **Authentication** | `Authorization: Bearer <token>` or `Sec-WebSocket-Protocol` | `Authorization: Bearer <token>` or `?ticket=<token>` |
| **Resume & Replay** | Control message `resume` with `afterSeq` | `Last-Event-ID` header or `?afterSeq=` query param |
| **Heartbeat** | WS Ping/Pong frames | `: keep-alive\n\n` comments every 15s |
| **Backpressure** | Connection drop on overflow | Connection drop on overflow (`event: overflow`) |

---

## Stream Ticket Authentication Flow

Browsers using standard `EventSource` cannot set custom HTTP headers. To prevent putting live session tokens in URLs, clients obtain a single-use stream ticket:

1. **Issue Ticket**: `POST /api/v1/stream/ticket` (authenticated via Bearer JWT)
- Returns: `{ "ticket": "eyJ...", "ttlSeconds": 60 }`
2. **Connect SSE**: `new EventSource('/api/v1/stream/sse?topics=portfolio,agent&ticket=eyJ...')`
3. Ticket is **single-use**, expires in **60 seconds**, and grants **read-only topic access**.

---

## Resumable Replay & Event Formatting

SSE events follow standard EventSource framing:

```http
id: 1042
event: deposit.received
data: {"amount": 100, "asset": "USDC", "status": "CONFIRMED"}

: keep-alive
```

- **`id:`**: Monotonic durable sequence number (`seq`).
- **`Last-Event-ID`**: When reconnecting, browsers automatically send `Last-Event-ID: 1042`. The server replays missed events from `seq > 1042`.
- **`event: replay_truncated`**: Emitted if `Last-Event-ID` is older than the server's data retention window.

---

## Example `curl` Usage

```bash
# 1. Get stream ticket
TICKET=$(curl -s -X POST http://localhost:3000/api/v1/stream/ticket \
-H "Authorization: Bearer $SESSION_TOKEN" | jq -r .ticket)

# 2. Connect to SSE stream
curl -N "http://localhost:3000/api/v1/stream/sse?topics=portfolio,transactions&ticket=$TICKET"
```
62 changes: 62 additions & 0 deletions docs/USER_WEBHOOKS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# User-Scoped Outbound Webhooks (#368)

User-scoped webhooks allow individual NeuroWealth account holders to register HTTPS endpoints to receive real-time HTTP POST notifications when events occur on their account (or permitted child sub-accounts).

## Overview & Architecture

Unlike operator-scoped webhooks (which fan out to system-wide operator endpoints with full domain payloads), **user-scoped webhooks deliver the exact redacted per-user projection** computed for the real-time stream.

Key capabilities:
- **Per-User Signing Secrets**: Each endpoint receives a unique HMAC secret (`whsec_...`) shown **only once** upon creation or secret rotation.
- **Event & Topic Scoping**: Endpoints can filter by specific domain events (`events: ["deposit.received", "agent.rebalanced"]`) or topic scopes (`topicScope: ["portfolio", "transactions"]`).
- **Server-side Filter Predicates**: Supports optional validated filter JSON predicates evaluated before delivery enqueueing (e.g. only `WITHDRAWAL` transactions over $100).
- **Idempotency & Replay**: Deliveries use `@@unique([endpointId, userEventSeq])` based on the user's durable stream sequence (`seq`). Endpoints can request replay via `POST /api/v1/webhooks/endpoints/:id/replay?afterSeq=`.
- **SSRF Protection**: Endpoint URLs must use `https://` and are validated against private, loopback, and link-local IP ranges.
- **Auto-Disabling**: After 5 consecutive delivery failures, an endpoint is marked `DISABLED_BAD_ENDPOINT` and a `webhook.endpoint_disabled` event is emitted.

---

## Signature Verification

Deliveries include an `X-NeuroWealth-Signature` header in the format:
```http
X-NeuroWealth-Signature: t=1700000000,v1=6a3f9e...
```

To verify the signature on your server:
1. Extract timestamp `t` and signature `v1`.
2. Compute HMAC-SHA256 over `${timestamp}.${rawBody}` using your endpoint secret.
3. Compare the computed hex digest against `v1`.

### Example Node.js Verification

```javascript
const crypto = require('crypto');

function verifyWebhook(rawBody, signatureHeader, secret) {
const parts = Object.fromEntries(signatureHeader.split(',').map(p => p.split('=')));
const timestamp = parts.t;
const expectedSig = parts.v1;

const signedPayload = `${timestamp}.${rawBody}`;
const actualSig = crypto.createHmac('sha256', secret).update(signedPayload).digest('hex');

return crypto.timingSafeEqual(Buffer.from(expectedSig), Buffer.from(actualSig));
}
```

---

## Management Endpoints

| Method | Endpoint | Description |
| :--- | :--- | :--- |
| `POST` | `/api/v1/webhooks/endpoints` | Register a new endpoint (returns secret once) |
| `GET` | `/api/v1/webhooks/endpoints` | List caller's webhook endpoints |
| `GET` | `/api/v1/webhooks/endpoints/:id` | Get endpoint details |
| `PATCH` | `/api/v1/webhooks/endpoints/:id` | Update URL, events, filterJson, or status |
| `DELETE` | `/api/v1/webhooks/endpoints/:id` | Delete endpoint |
| `POST` | `/api/v1/webhooks/endpoints/:id/rotate-secret` | Rotate HMAC secret (returns new secret once) |
| `POST` | `/api/v1/webhooks/endpoints/:id/test` | Dispatch `webhook.test` ping event |
| `POST` | `/api/v1/webhooks/endpoints/:id/replay?afterSeq=` | Re-enqueue events from stream |
| `GET` | `/api/v1/webhooks/endpoints/:id/deliveries` | View recent delivery history |
Loading
Loading