Repository navigation
Conversation
Every organization is told when a workflow that runs on its own fails 5 runs in a row: by default the owners get an email, with no settings row needed, so existing organizations are covered too. - A trigger on workflow_executions queues the workflow whenever a run ends, which covers every code path that finishes a run. - A once-a-minute internal cron (failure-alerts) derives the streak from run history, opens one incident per streak (unique open per workflow), and resolves it on the next automated success. - Delivery goes through an outbox with retries (1, 2, 5, 15, 30 min, then hourly for 24 h) and error classes. A message still undelivered after 15 minutes (2 for PagerDuty), or failed for good, goes to the backup channels and to email. - Dead channels are paused and the org is told once. A weekly silent check finds them without sending anything. - Free plan: 100 alert emails a month; only accepted emails count. - Rollout via FAILURE_ALERTS_ROLLOUT (all on staging and PR envs, off in prod). Only failures after an org is switched on count, and owners of orgs with earlier runs get a one-time notice listing what is failing. sendEmail now has a detailed variant that reports the provider status, and backup-channel outcomes carry the HTTP status and whether to retry.
…history Settings, Notifications gets a Workflow failure notifications card: alert after N failed runs in a row (default 5) with an optional time limit, daily reminder and recovery switches, member email list (wallet members without a verified email shown but not selectable), and Slack, Discord, Telegram and PagerDuty channels, each set to Always or Only as a backup. Every channel has a Test button, also before it is saved; a new channel shows Untested until a test passes. The PagerDuty test defaults to a change event that pages nobody. The email test goes to you, or to everyone ticked. Tests are rate limited, never count toward the Free plan's emails, and are audited. A Recent notifications card lists every alert, reminder, recovery and backup with where it went and what happened, filterable by channel, 50 at a time, with a CSV export. A workflow's Properties get a Notifications on fail section: the org default it inherits, a failing-streak line, Custom (per field, so untouched fields keep following the org) and a time-boxed mute of up to 30 days. Owners and admins of an org that nobody could be told about get a dark red banner linking to the settings, and a Workflows failing item in the user menu. Alert links open the workflow on its Runs tab. The channel editor reuses the PagerDuty node's service picker, the connection dialogs and the integration icons.
- The evaluator's queue lease went through a raw query as a JS Date, which the sql template sends as its local-time display string and reads back in the process's time zone, so outside UTC the lease never matched and the release query failed. The lease now stays in Postgres's own text form and the recheck time is sent as ISO. - Counting starts ten minutes before the evaluator first sees an org, so the failed run that brought the workflow to it is counted. - A failed PagerDuty page sends the backup email even to people who already have the alert by email: it means nobody on call was woken. - One-time notices (notifications switched on, a channel paused) no longer count toward the Free plan's monthly alert emails.
- History rows say which channel a pause notice is about and why a backup was sent; the CSV export gains a note column. - The paused-channel line no longer doubles the full stop, and a failed test no longer repeats the HTTP status the message already names. - The mute form's button is Mute alerts, so it is not confused with the Mute button on the failing banner, and the mute end shows date and time without seconds.
Reviewed from the user's side and cut down: - The settings card saves as you go, with a Saved indicator. A channel that was added and tested can no longer be lost by leaving without pressing Save at the bottom of a long card. - The card shows the threshold, who gets email and the channels. The time limit, daily reminder and recovery message move under More options; the time limit is a checkbox instead of an Off dropdown. - One Send me a test email button instead of a button plus a recipient picker. - A paused channel shows one Paused badge, and the general untested warning and the delivery paragraph are gone; How delivery works expands on demand. - Telegram explains how to find a chat ID. - In Properties, the Org default / Custom / Muted tabs are replaced by a summary and two buttons: Mute (1 hour, 24 hours, 7 days, until it works again, or a date, one click) and Customize. A mute that has ended no longer shows as muted.
Correctness - Incidents of workflows that are paused, deleted or switched to a manual trigger are closed by a sweep each cycle; before, nothing noticed and reminders and PagerDuty pages went on. - Every way an incident closes resolves its PagerDuty page on the same key, whether or not recovery messages are on; pages still in flight get the resolve slightly later. - An incident and its alert rows are written in one transaction, and an open incident with no alert after two minutes is re-queued. - Grouped alerts use the org's backup channels. - Queued messages carry a unique dedup key, since a unique index over a nullable incident id let grouped alerts and notices duplicate. - Rows are sent in small batches and each row's clock restarts before its send, so a slow batch is never taken over and sent twice. - A refused email key on our side keeps retrying instead of failing. - A PagerDuty service in maintenance counts as not delivered, so the backups fire, without pausing the channel; pages use the node's own event builder. - Settings saves run in one transaction under a per-org lock; channels are unique per destination, and a duplicate is refused with a message. - A test of a saved channel uses that channel's stored connection. - OAuth tokens are held to their organization like API keys; the workflow route authenticates before looking the workflow up. - Deleting a connection no longer deletes its channels silently: the next send pauses the channel and tells the org. - The launch notice is claimed on its own flag, so orgs that saved settings first still get it. - The cron route answers 500 when a stage failed; the unread heartbeat table is gone. Old history is pruned after 90 days. Reuse - One org-recipients module for quota warnings and failure alerts, the existing getOrgRole, a shared emailLayout in lib/email, the PagerDuty change-event path and Slack retry rule shared with the plugins, view types and labels shared between server and browser, Notice moved to components/ui and used by PagerDuty fields and this feature, roleLabel, timeAgo and truncateAddress. UI and copy - Plain-language provider errors, org name first in chat, settings link in every alert email, Open the workflow link, RPC and gas hints. - Failing now list, error states with Retry, phone layouts for channel and history rows, banner link always visible, unshared connections explained, Undo after removing a channel, focus kept when Mute or Customize opens, and accessibility fixes.
- No alert, reminder or backup for an incident that already closed; every attempted PagerDuty page is resolved, and closing an incident is one transaction - One delivery budget per cycle, checked before every row; unreached rows go back to the queue; retry and delivery times use the time of the send - Reminders oldest first, skipped incidents move on; inactive-incident sweep filtered in SQL; lost-alert requeue ordered and rollout-aware - Trigger rules reuse the workflow store; shared scope and recipient helpers - A channel whose connection was deleted no longer blocks saving other settings; Undo puts a paused channel back paused; removals run before updates; duplicate channels answer 409 - Failure notifications in the phone workflow sheet; Escape closes Mute and Customize; "Test failed" badge - Consistent wording, history labels matching the emails, shared date format with time zone, warning colour defined once, toasts replace each other, plainer Telegram help, grouped email links to all failing workflows - Indexes: drop a redundant one, add incidents(workflow_id), backup index without delivered rows
…fications in analytics - Per-workflow "Pause this workflow when it reaches the limit" in the Customize panel. When its alert goes out the workflow is turned off (the Enable switch's column), the alert says so, and the pause is audited. The incident stays open while the workflow stays off; if it keeps failing after being turned back on it is paused again and people are told. A mute does not stop a pause from being announced. There is no org-wide equivalent. - Analytics runs list: failure-notification deliveries as a third source, "System notifications", included by default, filtered by status, search and project, excluded by network, gas and duration filters; KPI cards and charts still count runs only. MCP list_executions and API docs updated. - Message kind labels and history notes shared from view-types.
…r saves - Alerts is its own settings page, readable by every member: When to notify (rules with their own Save), How to notify (one card per method: icon, target, status, Test, Save; email cannot be removed), Failing now, Recent notifications; Notifications keeps the Execution digest and a pointer - A new or changed Telegram, Discord, Slack or PagerDuty target is checked before it can be saved; channels are checked daily and a paused channel that works again comes back on its own; a single chat method with nobody reachable by email gets a backup suggestion - Org-wide pause for maintenance (up to 7 days, with a reason): people are told when it starts and ends, nothing alerts or auto-pauses meanwhile, and what kept failing is reported on resume - Scheduled workflows that do not start within 10 minutes of their cron or interval are reported, and close with a running-again message; a reaper watchdog pages our on-call if alerts stop being processed - Alerts read as urgent: "Action needed: X failed 5 times in a row", the runs it succeeded before, the last error and direct links to the failing workflow and its runs; email, chat and PagerDuty share one text; tests send this real format marked [Test]; "See what people get" preview - Pause fixes from review round 3: pausing a workflow that is already failing, a resume recorded by a trigger on workflows.enabled so a later manual pause closes the incident, one transaction for pause + claim + message - Analytics: notification rows keep their own labels, never count in totals, facets or KPIs; compound cursor so paging never skips tied rows; faster project and search filters
Server - Grouped alerts only for workflows that follow the org's people and methods; PagerDuty group field and warning severity for missed runs - "Until it works" mutes end after 30 days; stale saves refused (version, 409) so two admins cannot undo each other's changes - Org pause still turns off workflows set to pause at their limit and holds the message until the pause ends; extending a pause is race-safe - Free plan: the last 10 emails kept for new failures, warnings at 80 and 100, missed runs never counted; reachability ignores methods whose connection is gone and a used-up email cap - Missed runs: schedule-aware grace (2x interval, 2 to 10 min), due check before loading settings, rotating scan, scheduler outages page our team first and reach customers after 30 min, "started again but failed" - Wallet out of funds: two shortfalls in a row (chain or pre-send check) alert at once with a fund-the-wallet next step - Telegram chat ID format checked on save; Slack's daily check no longer re-enables a paused channel; history of open alerts is never pruned; exhaustive method switches; per-cycle limits for the launch scan - Messages: one vocabulary (Alerts), reminder N of 7 with a last-reminder line, amber Check for missed runs, per-workflow errors in grouped alerts, recovery with duration, correct gas hints, email-limit warning email - Notifications status exposes alertsEnabled; API and MCP list alert rows only when asked; shared delivery state labels UI - Alerts page: Status card (all quiet, failing, paused, unreachable, single method, email limit), rules, collapsed method rows with honest states and last checked/delivered, workflows with their own settings, sent alerts - Hidden when alerts are off for the org; stale-save Reload; unsaved changes guard; chat ID validation; org pause in the workflow panel; width fixes Also: comments trimmed across the branch's server code
zdumitru
added
deploy-pr-environment
run-e2e-tests-ephemeral
no-issue-required
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Analytics lists runs only again, so the card is titled Workflow Runs and the list_executions source takes workflow and direct, as on staging.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
…n, saves re-queue only on rule changes - A wallet member's step-up inbox no longer leaks: the Sent alerts To filter matches what the log shows, and the Test reply and stored provider text are masked the same way (one shared helper). - A refused incident closes only on a run that was really accepted (shared acceptedRunSql), not on a phantom, pending or platform-failed row. - The launch notice is claimed and queued in one transaction. - The Analytics "Alert sent" chip uses the run that opened the incident, stored once in the new opening_execution_id column (migration 0167, with an index). - Org and workflow saves re-queue evaluation only when something the evaluator reads changed (threshold, time limit, mute, auto-pause), not on recipient or method edits. - Docs describe the runs list's optional alert object.
Workflow panel: - Saves go one at a time, each on the latest version; a draft is sent once. - An edit is sent when the panel closes; an unfinished or failed one says it was not kept. - "Use org rules" can no longer be undone by Run, Save or closing the panel, and a failed switch says so. - The tab changes only when the rules in use change. - A failed save shows a toast and "Not saved", with Try again only for server or network failures. - Alert edits keep their own unsaved flag: the Save dot shows it, Run's save check does not, so an alert save never hides an unsaved canvas. - A Test result applies to the method list as it is now. Org page: - A newer save moves only the fields the user has not edited. - An older reply is ignored. - An invalid value shows "Not saved" instead of "Saved". Cleanup: dead Save props removed, one warning palette, one refusal helper.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| const canManage = data.canManage; | ||
| const rulesBlocked = | ||
| canManage && | ||
| rules && |
A "not running" incident no longer ends on a phantom placeholder or a queued
row no worker has claimed, and such rows no longer hide a schedule that stopped
starting runs from the missed scan. A run our platform failed still counts as
started: the schedule fired, and counting it keeps a platform outage from
telling every customer their schedules stopped. latestRunSince takes a named
mode ("started" or "accepted") instead of a positional flag.
Sent alerts masks wallet members' step-up inboxes by looking users up by step_up_email on every page and To filter; the column had no index. Partial on step_up_email IS NOT NULL, so it holds only wallet members with an inbox. Marked @requires-db-prep: users is a hot table, so an operator creates it CONCURRENTLY on staging and prod before merge and the migration is a no-op there.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
…s gets alerts Silencing a workflow can no longer go unnoticed: - A mute lasts at most 7 days, "until it works again" included, and needs a reason (3 to 200 characters). Existing mutes are capped by migration 0169. - Muting, changing, unmuting and the end of a mute send a notice to the people who got the workflow's alerts before the change plus every owner (or reachable admins when no owner can get email), queued in the same transaction, shown in Sent alerts. At most 6 mute changes per workflow per hour. - When a mute ends, the notice gives the runs that failed meanwhile, and a workflow still failing is alerted again at once. - People taken off a recipient list are told, also when a list leaves the default, and so are the owners. - Separate audit actions for mutes and unmutes, with before and after. Recipients and defaults: - New orgs alert after 3 failed runs and email the owners. - Every list keeps an owner or admin who can get email; people who leave drop off, and a list with nobody left goes back to the owners. - Who paused alerts is shown with their sign-in account. Sent alerts can be filtered by workflow name or ID. The workflow and history APIs are documented in docs/api/failure-alerts.md.
…le tabs - The workflow panel has one Mute area outside the Org rules / Custom rules tabs: a reason is required, and the form says who will be told. Muted, it shows who, until when and why, with Unmute and Change or extend. - Overview's Needs attention has Setup and Workflows subsections in one format; every active mute is listed (failing ones first, in red) with an Unmute button. - Org rules and Custom rules read "Who gets alerts" then "When to alert", with the reminder and recovery settings; "Disable workflow" wording. - The last owner or admin who can get email cannot be unticked. - Sent alerts: workflow filter, mute notes, clearer By column on phones. - Alerts page copy: scope note, workflows can use their own setup, Status description, no duplicate "Alerts reach" or "Last alert delivered" lines. - The settings column keeps its width when a tab scrolls.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
A mute and a success created in the same instant raced the database clock against the test runner's, so "ends only on a success after it began" could miss in CI. The tests now start the mute minutes earlier.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Reserving it on phones shifted every settings page 10px off centre, where scrollbars overlay anyway. Tablets and desktops keep it, so a tab that scrolls is no narrower than one that does not.
…mid-word Sent alerts: - Four columns (When, What, Sent to, Result): who acted shows under the kind, the method icon sits before the address. - "Alerts only | People's actions | Everything" replaces the pause-notices chip; People's actions (mutes, pauses, removals, tests) is a new history scope, documented in docs/api/failure-alerts.md. - "All delivered", or "N not delivered" as a button that shows only those. - Mute and pause rows always give a reason or "No reason given"; mute notes read "Mute ran out · no failures while muted" and show the end in local time. - An empty result says so inside the table, with Clear filters. - Relative times with the exact time on hover, the viewer's time zone in the header, a sticky header, one toolbar row; "Alert" is now "New failure". Emails break only before "@" (never mid-word or at a hyphen) in Sent alerts, the Email card and both recipient pickers, which now show the name with the email below it.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Collaborator
Author
|
Since the review, e36ec44, 09b5aca, 275eb18, 2dfb1f8 and 7e4510f address the comments above. Later commits add follow-ups from the owner. A mute now needs a reason, lasts at most 7 days, is announced to the people who would get alerts, and shows in Needs attention. There is one mute control, outside the rule tabs. An owner or admin always gets alerts (f705e1e, 9a21bc5). The Sent alerts log is more compact (e80f622). Migration 0168 is @requires-db-prep: the staging index has to exist before db-prep-check passes. |
… the org rules - Email different people opens the same Email card as Settings > Alerts. - The workflow's connection picker lists the viewer's unshared connections greyed out, as Settings does. - Testing a saved workflow method reloads its health; drafts stay. - The Org rules tab drops "Change them in Settings"; the footer link and the unreachable notice both open Setup.
…od save - page above 100000 is refused with 400 instead of overflowing the offset. - After a Test, methods' health is merged only from a reply of the same version, so an older reply cannot bring back a method's old target.
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
…r digest - Rows of orgs outside this process's rollout wait and retry instead of being skipped, so a widening rollout mid-deploy loses no alert. - Mute ending counts runs the way the streak does; a deleted workflow's or deactivated org's mute ends without a notice. - Mute and pause ends a few minutes past 7 days are trimmed, not refused; ends without a zone are read as UTC. - Names, reasons and subjects in every message are one line with no control or bidi characters; mute reasons are cleaned when saved. - Daily reminders for org-rule workflows go out as one message per person, like first alerts. - Mute errors show in the mute panel; a JSON body that is not an object gets 400; an edit made just before a reload is still saved. - Emails wrap only before "@" and are never cut off; touch targets, landscape banner text and several labels follow the UX review. - Migrations folded into 0166 (alert tables) and 0167 (users index). - Org-level routes documented; CSV, wallet and name helpers reused.
- Daily reminders: reminders due later the same day join the org's one message; fewer than 3 left after claiming go out one by one; claims in one lock order; grouped and single reminders fail separately. - Digest reads as a daily reminder: counts add up, not-running ones are listed first, it says how to stop it, no "shared cause" line. - A mute ending on refused runs no longer says the workflow works. - Prune keeps grouped rows with two indexable conditions; rollout-held rows move in one update and expire as outside the rollout. - Overview says "stopped working" for a method that failed but is not paused; a muted row only adds "Still failing" when it is. - A reload during the autosave delay asks first; one refresh path with one failure message; the reload send shares the save's body. - Emails keep their brackets on the same line; tablet banner uses the short text; mute buttons are 44px tall on touch; clearer clock-ahead refusal and mute helper text. - parseEndTime in its own module; nameOf used in the pause route and test sends; API docs for optional version and the reminder flags.
- Reminders due later the same day join only when that makes a shared message; none goes out early on its own. A claim skips an incident closed since it was read; a group that drops under 3 gives every claim back and the due ones go out alone. - A save leaving out remindDaily or notifyRecovery keeps them as they are, like channels. - Leaving with an unsaved edit starts the save while the page is alive (keepalive), so leaving or staying it lands. - The panel says a muted workflow is still turned off at its limit; muted Overview rows are yellow; the digest says how to stop it and its plain text reads "In <org>, ...". - Email pattern without nested repeats; prune checks incidentIds is an array; nameOf beside pauseActorName; shared HOUR_MS and the digest window in constants; own toast id for refresh failures.
- Digest reads "N workflows are still failing" and names the "Remind once a day" setting; Overview lists unmuted failures before muted. - The reminder pull-in query returns only rows not yet due. - A page going away sends a draft still queued behind another save, and never a draft that already saved or failed. - The pause form uses the server's limits and shared time units; the audit leaves out reminder flags a save did not send. - Tests: the digest wording, reminder flags kept when left out, and that the save leaves before the "Leave site?" answer.
- Leaving with an edit queued behind a running save sends it at once while the page is alive (a stale version is refused, never overwrites); otherwise the save starts as before. - The last saved draft is forgotten when stored rules are replaced, so a page going away never skips a real change. - The leave test checks the saved value, staying and leaving; shared HOUR_MS in the workflow panel.
Collaborator
Author
|
Changes since the last summary (e80f622 to 038a873):
Checks on 038a873: 47 browser tests, 327 unit tests and 196 DB tests pass locally; lint, types, API docs and schema drift are clean. |
Contributor
PR Environment DeployedYour PR environment has been deployed! Environment Details:
Components:
The environment will be automatically cleaned up when this PR is closed or merged. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: workflow failure alerts. Review feedback is addressed (replies on each comment); staging is merged in; the PR environment runs this branch on a fresh database.
What it does
Every organization is told when a workflow that runs on its own (Schedule, Event, Block, Webhook, Transfer, Pyth Price; never Manual) keeps failing:
Changes outside the alerts code
lib/workflow/refused-runs.ts: every run refused before it starts is recorded the same way (skipped, not billable, category and reason), on the internal dispatch, webhook (plan limit and paid feature, which recorded nothing before), execute, MCP call, Pyth and executor paths. Refusals of one workflow and reason share an hourly row with a refusal count; under a flood the row is updated at most once a minute. Billing decisions, HTTP responses and metrics are unchanged.skippedinstead oferror, like the executor's; the 402 bodies are unchanged.components/navigation-guard.tsx: an unsaved-changes guard for router navigation, Back, org switch and New Workflow (used by the Alerts page).app/settings/(hub)/layout.tsxpasses the user's orgs with alerts on, so the Alerts menu item draws with the rest instead of after a request.SectionHeadergains an optionalwidedescription./api/notifications/statusreturnsalertsPausedUntiland the nobody-reachable flag to every member (the failing-workflows notice stays with owners and admins).components/app-banner-slot.tsx,app-banner.tsx,billing/quota-banner.tsx) stacks an alerts banner above a billing banner and offsets the page by both.lib/analytics/alert-sent.tsmarks those runs on the page being listed (one query per page, only failed workflow runs); the runs list, filters, numbers, API and MCP tool are otherwise as on staging.How it is built
0166_workflow_failure_alerts(dated after staging's 0165):alert_*tables, prune indexes, and two triggers (queue a workflow when a run ends in success, error or skipped; record when a paused workflow is turned back on);0167(the run that opened an alert, for the Analytics chip);0168(partial index onusers.step_up_email,@requires-db-prep: create it CONCURRENTLY on staging and adddb-prepped-stagingbefore merge);0169(default threshold 3, who paused and when a mute started, existing mutes capped at 7 days).lock_timeoutis set; no existing table changes except the new index./api/internal/failure-alerts: evaluate, missed schedules, deliver with retries and backups, health checks, prune. Only evaluate and deliver failures fail the job.FAILURE_ALERTS_ROLLOUT(off/all/orgs:<ids>/percent:<n>). Outside the rollout every alert route answers 404, nothing is delivered or checked, and the queue is cleared each cycle.Verified
.env; 477 alert, notification, analytics and settings tests after the latest changes.failure-alerts.test.ts(15 tests) on a production build: the three tabs, rules saving themselves with Undo, a refused method, the Status switch, Sent alerts, the workflow tabs, the Analytics chip opening Sent alerts for its workflow, and no sideways scroll on every tab at 390, 768 and 1280 (earlier also 1024 and 1920). Also checked at 1280 and 390: stacked banners while paused, the Status card states, Sent alerts table, filters and paging, Preview per method, Add method keeping the editor in place, and the Settings menu drawing all items at once. Owners and admins can change every setting (route test per write); plain members get 403.drizzle-kit generatereports no drift.Not done yet