Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/v1.5/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This tree is **v1.4**. Switch versions in the sidebar for older releases.
- [Navigation, search, and alerts](./core-concepts/navigation-search-notifications) — Find records and use the in-app inbox
- [Installation](./getting-started/installation) — Compose, Helm, Kustomize, from source. First boot needs `NEXTAUTH_SECRET` and `ENCRYPTION_KEY`.
- [Troubleshooting](./troubleshooting) — Compose, database, auth, paging
- [Notifications](./administration/notifications) — How someone actually gets paged (no voice).
- [Notifications](./administration/notifications) — How someone actually gets paged (no voice), including user-controlled Quiet Hours.
- [Incidents](./core-concepts/incidents)
- [Escalation policies](./core-concepts/escalation-policies)
- [On-call schedules](./core-concepts/schedules)
Expand Down
32 changes: 28 additions & 4 deletions docs/v1.5/administration/notifications.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ valid incident recipient
+ enabled workspace provider
+ enabled user preference and contact/device data
+ escalation or service event
+ user Quiet Hours policy when explicitly enabled
→ notification attempt and history
```

Expand All @@ -23,7 +24,7 @@ Saving one layer does not verify the whole path. Test every production recipient
## Permissions and settings

- An application **Admin** configures workspace providers in **Settings → Notification Providers**.
- Each user configures personal Email, SMS, Push, and WhatsApp preferences under **Settings → Notifications**.
- Each user configures personal Email, SMS, Push, WhatsApp, and Quiet Hours preferences under **Settings → Profile & Preferences → Notification Preferences**.
- Admins/Responders configure service-level Slack and webhook events under **Service → Settings**.
- Policy administration is Admin-only. New steps in the current v1.4 UI inherit user preferences; the UI does not expose new per-step channel overrides.
- Signed-in users can open **Settings → Notification History**; access to operational data should still be governed by deployment policy.
Expand Down Expand Up @@ -55,10 +56,32 @@ It attempts channels in order and normally stops after the first successful non-

This is ordered fallback, not guaranteed fan-out to every enabled channel. An in-app notification is created separately even when no external channel is available.

Stored escalation-channel data, when present, is intersected with the user's available channels. If the intersection is empty, the implementation falls back to the user's available preferences rather than dropping the page.
Stored escalation-channel data, when present, is intersected with the user's available channels. If the intersection is empty, the implementation falls back to the user's available preferences rather than dropping the page. Quiet Hours filtering remains in force during fallback, so an intentionally suppressed LOW-urgency channel is not reintroduced by the fallback path.

Service-level Slack/webhook/email/SMS/push/WhatsApp notifications are a separate path selected by service event settings. Avoid configuring duplicate paths until you have observed their combined behavior.

## Quiet Hours

Quiet Hours is a **personal, explicit opt-in** notification policy. It is **off by default for both existing and new users**. OpsKnight never enables it automatically during an upgrade or when an account is created.

Users configure it at **Settings → Profile & Preferences → Notification Preferences → Quiet Hours**. When enabled, they can choose:

- start time;
- end time; and
- whether Saturday and Sunday are quiet all day.

The configured times are evaluated in the user's profile timezone. The initial editable schedule is 18:00–08:00 with all-day weekends, but it has no effect until the user turns Quiet Hours on.

During an active Quiet Hours window:

- **LOW urgency**: Push, SMS, and WhatsApp are suppressed;
- **Email and in-app** notifications remain available; and
- **MEDIUM and HIGH urgency** bypass Quiet Hours and continue paging normally.

Suppression is intentional policy behavior, not a provider failure. OpsKnight does not create a false failed-delivery result merely because a channel was excluded by Quiet Hours. Invalid timezone/window configuration fails open rather than silently suppressing a page.

For incident-response safety, do not use Quiet Hours as a substitute for escalation-policy design, schedule coverage, or urgency mapping. If a responder must always receive a class of alert, classify and route it appropriately instead of relying on a personal LOW-urgency policy.

## Configure email

Choose exactly the provider intended for workspace email and supply:
Expand Down Expand Up @@ -157,6 +180,7 @@ For every on-call user:
- [ ] at least one external channel is enabled and usable;
- [ ] phone number is E.164 when SMS/WhatsApp is enabled;
- [ ] push is registered on the intended device;
- [ ] Quiet Hours is understood and intentionally configured if enabled;
- [ ] Team notification participation is enabled for team-targeted paging;
- [ ] a direct test policy reaches the user;
- [ ] the user can open and acknowledge the incident.
Expand All @@ -181,7 +205,7 @@ There is no manual Retry button in the history page. Correct the provider or rec

### No notification record

Check that the incident actually targeted the user/team/schedule, the policy ran, assignment and lifecycle event are correct, and the user was eligible. Review the incident timeline and escalation state.
Check that the incident actually targeted the user/team/schedule, the policy ran, assignment and lifecycle event are correct, and the user was eligible. For LOW urgency, also check whether the user explicitly enabled Quiet Hours and the channel was intentionally suppressed. Review the incident timeline and escalation state.

### `FAILED` email

Expand All @@ -193,7 +217,7 @@ For Twilio, check credentials, sender capability, trial verification, regional p

### Push does not arrive

Check HTTPS, browser support and permission, service worker `/sw.js`, VAPID public/private pairing, saved subscription, user preference, OS/browser background restrictions, and Test Push.
Check HTTPS, browser support and permission, service worker `/sw.js`, VAPID public/private pairing, saved subscription, user preference, Quiet Hours for LOW urgency, OS/browser background restrictions, and Test Push.

### Slack or webhook fails

Expand Down
16 changes: 10 additions & 6 deletions docs/v1.5/core-concepts/escalation-policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ Only an application **Admin** can create, change, reorder, or delete policies an
3. It resolves the step target to one or more users, assigns an unassigned incident to the target, and sends notifications.
4. It schedules the next step using that next step's delay.
5. Acknowledging, resolving, snoozing, or suppressing the incident stops or pauses further escalation according to the incident lifecycle.
6. If a target is invalid or resolves to no users, the timeline records the failure and OpsKnight advances to the next step. After the last step, escalation is complete.
6. If a target is invalid or resolves to no users, the timeline records the failure and OpsKnight advances when another step exists. If the final step cannot resolve a valid recipient, the escalation ends in `FAILED` rather than being reported as successfully completed.

Delays mean “wait before this step,” not “wait after the previous notification.” A zero delay executes the step immediately.

Policies do not repeat in v1.4. After the last step is exhausted, the escalation is complete.
Policies do not repeat in v1.4. A policy that executes through its valid steps can finish as completed; terminal routing failures remain failed so operators can distinguish exhaustion from successful execution.

## Target types

Expand Down Expand Up @@ -49,6 +49,8 @@ The policy can be saved with no steps, but it cannot page anyone until at least

New steps created in the current v1.4 policy interface do not expose a per-step channel selector. They use each resolved user's enabled notification preferences and the configured workspace providers. Although the data model supports stored step-channel overrides, do not depend on an undocumented database-level configuration as a public workflow.

Personal Quiet Hours is a separate recipient policy. It is off by default and must be explicitly enabled by the user. When active, it can suppress LOW-urgency Push, SMS, and WhatsApp delivery for that recipient; Email and in-app remain available, and MEDIUM/HIGH urgency bypasses Quiet Hours. Fallback does not reintroduce a channel that Quiet Hours intentionally suppressed.

Likewise, the current add-step interface does not expose the Team Lead-only toggle. Existing lead-only steps can execute and are labeled in the policy view, but new policy design should not depend on setting that flag through the v1.4 UI.

## Design a resilient policy
Expand All @@ -70,6 +72,7 @@ For every policy:
- monitor schedule end dates and coverage gaps;
- ensure each target has at least one configured delivery channel;
- use descriptions that distinguish critical and non-critical paths;
- verify urgency mapping is appropriate for alerts that must always page regardless of personal Quiet Hours; and
- retest after membership, schedule, provider, or policy changes.

## Reorder and edit safely
Expand Down Expand Up @@ -102,25 +105,26 @@ Use a non-production service or coordinated test window:
4. Allow the next step to execute and verify its target and timing.
5. Acknowledge and confirm no later step runs.
6. Repeat during a schedule override and a known coverage edge.
7. Resolve and remove test artifacts according to retention policy.
7. For a LOW-urgency test, explicitly verify Quiet Hours behavior only on a user who chose to enable it.
8. Resolve and remove test artifacts according to retention policy.

## Troubleshooting

### Escalation does not start

Confirm the service has this policy, the incident is Open, the policy has steps, and the incident escalation state is not already completed or paused.
Confirm the service has this policy, the incident is Open, the policy has steps, and the incident escalation state is not already completed, failed, or paused.

### A step resolves to no users

- User: confirm the account exists and is active.
- Team: confirm eligible members have team notifications enabled; for a lead-only legacy step, configure a Team Lead and enable their team notifications.
- Schedule: inspect effective coverage at the execution time, including timezone, restrictions, gaps, priority, and overrides.

OpsKnight records the problem in the incident timeline and advances when another step exists.
OpsKnight records the problem in the incident timeline and advances when another step exists. If the final step has an invalid target or no eligible users, the terminal escalation state remains `FAILED` so the routing problem is visible instead of being mislabeled `COMPLETED`.

### A target is correct but receives no message

Check the user's notification preferences, contact/device data, provider settings, notification history, and system logs. A policy target is not proof that a delivery provider accepted the message.
Check the user's notification preferences, contact/device data, Quiet Hours state for LOW urgency, provider settings, notification history, and system logs. A policy target is not proof that a delivery provider accepted the message.

### Later steps continue after response

Expand Down
18 changes: 12 additions & 6 deletions docs/v1.5/core-concepts/users.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,26 +86,32 @@ Each user can open **Settings → Profile** to manage supported profile fields,

OpsKnight provides a zero-latency, local `@dicebear` vector avatar generator at `/api/avatar`. Users can select from 15 curated SVG style presets (including `bottts`, `shapes`, `initials`, `personas`, `identicon`, `avataaars`, `thumbs`, `lorelei`, `notionists`, `open-peeps`, `micah`, `miniavs`, `pixel-art`, `rings`, and `glass`) with gender-appropriate styling. Avatar SVGs are rendered locally without third-party network dependencies, cached immutably with `Cache-Control: public, max-age=31536000, immutable`, and served with strict SVG sandbox headers (`Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox`).

Timezone affects how the application displays dates for that user. Schedule calculation remains authoritative in each schedule's timezone.
Timezone affects how the application displays dates for that user. Schedule calculation remains authoritative in each schedule's timezone. Quiet Hours also evaluates its configured times in this profile timezone.

## Notification preferences

Open **Settings → Notifications** to enable supported user channels:
Open **Settings → Profile & Preferences → Notification Preferences** to configure personal notification behavior:

- Email;
- SMS, with a phone number;
- Push, with a registered supported device and provider;
- WhatsApp, with a phone number in E.164 format.
- WhatsApp, with a phone number in E.164 format; and
- **Quiet Hours**, an optional LOW-urgency suppression policy.

Quiet Hours is **off by default** for both existing and new users. OpsKnight does not silently mute paging after an upgrade or account creation. A user must explicitly enable Quiet Hours before it can suppress any delivery channel.

When enabled, the user chooses a start time, end time, and whether weekends are quiet all day. Times use the user's profile timezone. During an active Quiet Hours window, only LOW-urgency Push, SMS, and WhatsApp delivery is suppressed. Email and in-app notifications remain available, and MEDIUM/HIGH urgency bypasses Quiet Hours entirely.

These switches express user preference; they do not configure workspace providers. Delivery requires all of the following:

1. the workspace provider is enabled and valid;
2. the user enabled the channel;
3. required contact/device data exists;
4. the escalation or service event selects or inherits that channel;
5. the provider accepts the message.
5. Quiet Hours does not intentionally suppress that LOW-urgency disruptive channel; and
6. the provider accepts the message.

Team paging also respects the membership's `Receive team notifications` setting. Test the full chain and review notification history rather than assuming a saved switch guarantees delivery.
Team paging also respects the membership's `Receive team notifications` setting. Test the full chain and review notification history rather than assuming a saved switch guarantees delivery. Quiet Hours suppression is an intentional policy decision and should not be interpreted as a provider delivery failure.

## Passwords and sessions

Expand Down Expand Up @@ -167,7 +173,7 @@ If the menu shows **Allow OIDC linking**, approval is not currently present. If

### A user is targeted but receives no page

Confirm account status, channel preference, contact/device data, team notification participation, workspace provider, and notification history.
Confirm account status, channel preference, contact/device data, team notification participation, Quiet Hours state for LOW urgency, workspace provider, and notification history.

### An account cannot be removed

Expand Down
8 changes: 5 additions & 3 deletions docs/v1.5/integrations/inbound-webhook-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ The middleware also accepts `Authorization: Token token=…`, `X-API-Key`, or th

The integration must exist, be enabled, and point to the intended service. The route validates the route-specific provider schema, but v1.4 does not separately compare the stored integration type with the route segment. Protect each integration ID/key pair and use only the URL generated for its intended sender. A workspace API key is not interchangeable with an integration key.

Authentication is resolved before OpsKnight consumes the database-backed per-integration rate limit. Requests using a nonexistent integration ID or an invalid integration key therefore do not create arbitrary per-integration rate-limit rows and do not consume the valid integration's quota. This limiter protects authenticated integration traffic; upstream proxy/WAF controls remain appropriate for broad unauthenticated abuse protection.

## Optional signature verification

The integration key is always required. When a signature secret is configured on the integration, OpsKnight also verifies the raw request according to the route's signature mode unless `INTEGRATION_VERIFY_SIGNATURES=false`. Verification is enabled by default; the environment override is a diagnostic escape hatch, not a production baseline.
Expand Down Expand Up @@ -90,16 +92,16 @@ Accepted event requests normally return HTTP 202 with a JSON result. Common fail
| 401 | Invalid key on legacy-handler routes, disabled integration on standardized-handler routes, or missing/invalid required signature. |
| 403 | Disabled integration on legacy-handler routes. |
| 404 | Integration not found. |
| 429 | Per-integration rate limit exceeded; honor `Retry-After`. |
| 429 | Per-integration rate limit exceeded after integration authentication; honor `Retry-After`. |
| 500 | Unexpected processing failure. |

Provider routes are being consolidated on the standardized handler, so some authentication failures currently differ between `400`, `401`, and `403`. Treat all three as non-retryable configuration/authentication failures; inspect the JSON message rather than branching only on one status.

Integration routes default to 100 requests per 60 seconds per integration when integration rate limiting is enabled. Rate-limit responses include remaining/reset information; successful routes do not guarantee every provider displays those headers.
Integration routes default to 100 requests per 60 seconds per authenticated integration when integration rate limiting is enabled. Invalid IDs and invalid integration keys are rejected before this database-backed quota is consumed. Rate-limit responses include remaining/reset information; successful routes do not guarantee every provider displays those headers.

## Production acceptance

For every configured provider, test a failure and recovery using the real upstream sender. Confirm the first event creates or updates the intended service incident, repeated events deduplicate, recovery resolves the same incident, invalid credentials are rejected, and responders receive the expected notification. Preserve the provider delivery record and OpsKnight event/timeline evidence for troubleshooting.
For every configured provider, test a failure and recovery using the real upstream sender. Confirm the first event creates or updates the intended service incident, repeated events deduplicate, recovery resolves the same incident, invalid credentials are rejected without consuming authenticated integration quota, and responders receive the expected notification. Preserve the provider delivery record and OpsKnight event/timeline evidence for troubleshooting.

## Related topics

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Quiet Hours is an explicit user opt-in. Existing and new users keep full paging by default.
ALTER TABLE "User"
ADD COLUMN "quietHoursEnabled" BOOLEAN NOT NULL DEFAULT false,
ADD COLUMN "quietHoursStartMinutes" INTEGER NOT NULL DEFAULT 1080,
ADD COLUMN "quietHoursEndMinutes" INTEGER NOT NULL DEFAULT 480,
ADD COLUMN "quietHoursWeekendAllDay" BOOLEAN NOT NULL DEFAULT true;

ALTER TABLE "User"
ADD CONSTRAINT "User_quietHoursStartMinutes_check"
CHECK ("quietHoursStartMinutes" >= 0 AND "quietHoursStartMinutes" < 1440),
ADD CONSTRAINT "User_quietHoursEndMinutes_check"
CHECK ("quietHoursEndMinutes" >= 0 AND "quietHoursEndMinutes" < 1440),
ADD CONSTRAINT "User_quietHours_window_check"
CHECK ("quietHoursStartMinutes" <> "quietHoursEndMinutes");
Loading
Loading