feat(validators): alarm the nominator-counts lane, and size both thresholds to the producer - #9311
Merged
Merged
Conversation
…sholds to the producer Two gaps in the alarm coverage over the two outputs of the poller's one Alpha scan. THE COUNTS LANE HAD NO WATCHDOG. #9302 gave it a D1 sink and metagraphed-infra #243 re-enabled its producer, but nothing watched it -- which is the exact shape of the failure that produced the lane's outage in the first place. Its writer targeted a Postgres that went away, nominator_count degraded to null or to a frozen 2026-08-02 mirror covering 564 of 1,031 validators, and no probe, no red check and no exception fired. The read path degrades so gracefully that the outage was invisible. nominator_positions -- the other output of the same scan -- got a watchdog in #9273 for precisely this reason; its sibling now has the matching one, same shape as neurons/positions/chain-detail before it: one MAX() read, a pure rule, a summary rather than a throw, one exception per stale tick. An EMPTY table alerts too, since that is the state in which every nominator_count is still coming from the frozen mirror. THE POSITIONS THRESHOLD UNDERCUT ITS OWN PRODUCER. It was 6 hours, chosen while that lane had no producer at all, on the reasoning that six hours was "several missed passes at any plausible cadence". The producer now feeding it runs on a 24h tick (VALIDATOR_NOMINATORS_POLL_SECS defaults to 24*3600, and one scan writes both tables), so a healthy lane presents an age anywhere in [0h, 24h+scan] and a 6h threshold would have alerted for roughly three quarters of every day. It has not fired yet only because the table is still empty and takes the no_rows branch instead; it would have started the moment the re-enabled producer posted. An alarm that always fires is one nobody reads. Both thresholds are now 30h -- one missed pass plus slack for the scan itself (~4 min at the measured ~3,100 rows/sec) and cron jitter. Derived from the cadence, not picked, and stated as such in both headers so the next cadence change has somewhere to land. Tests pin the regression directly: a capture from the middle of a 24h cycle must be quiet. Cron 19,49 collides with nothing in workers/config.ts and stays off the */5 raw-capture and */15 probe grids; the suite asserts both that uniqueness and that wrangler.jsonc actually declares the trigger, since dispatch keys on the literal string and an undeclared cron is silently dead code. Closes #9310 Part of #9146
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | c4b38da | Aug 03 2026, 08:55 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | c4b38da | Aug 03 2026, 08:55 PM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two gaps in the alarm coverage over the two outputs of the poller's one Alpha scan.
1. The counts lane had no watchdog
#9302 gave it a D1 sink and metagraphed-infra#243 re-enabled its producer, but nothing watched it — which is the exact shape of the failure that took the lane down in the first place. Its writer targeted a Postgres that went away,
nominator_countdegraded to null (or to a frozen 2026-08-02 mirror covering 564 of 1,031 validators), and no probe, no red check, no exception fired. The read path degrades so gracefully that the outage was invisible.nominator_positions— the other output of the same scan — got a watchdog in #9273 for precisely this reason. Its sibling now has the matching one: oneMAX()read, a pure rule, a summary rather than a throw, one exception per stale tick underwatchdog:validator-nominator-counts-staleness. An empty table alerts too, since that is the state in which everynominator_countis still coming from the frozen mirror or serving null outright.2. The positions threshold undercut its own producer
NOMINATOR_POSITIONS_STALENESS_THRESHOLD_MSwas 6 hours, chosen while that lane had no producer at all, on the reasoning that six hours was "several missed passes at any plausible cadence".The producer now feeding it runs on a 24h tick (
VALIDATOR_NOMINATORS_POLL_SECSdefaults to24 * 3600, and one scan writes both tables). A healthy lane therefore presents an age anywhere in[0h, 24h + scan], so a 6h threshold would have alerted for roughly three quarters of every day.It hasn't fired yet only because the table is still empty and takes the
no_rowsbranch instead — it would have started the moment the re-enabled producer posted its first batch. An alarm that always fires is one nobody reads, so I fixed it here rather than leaving a known pager-burner behind a change of mine that activates it.Both thresholds are now 30h: one missed pass plus slack for the scan (~4 min at the measured ~3,100 rows/sec) and cron jitter. Derived from the cadence rather than picked, and stated as such in both headers so the next cadence change has somewhere to land.
Tests
The regression is pinned directly — a capture from the middle of a 24h cycle must be quiet — on both lanes, so nobody re-tightens the threshold under its cadence without a red test. Four positions-suite fixtures that encoded the old 6h assumption were updated rather than deleted.
Cron
19,49collides with nothing inworkers/config.tsand stays off the*/5raw-capture and*/15probe grids. The suite asserts both that uniqueness and thatwrangler.jsoncactually declares the trigger — dispatch keys on the literal string, so an undeclared cron is silently dead code.Patch coverage verified by diff intersection: 0 uncovered lines, 0 uncovered branches across all four changed source files.
Closes #9310
Part of #9146