Skip to content

feat(validators): alarm the nominator-counts lane, and size both thresholds to the producer - #9311

Merged
JSONbored merged 1 commit into
mainfrom
feat/validator-nominator-counts-watchdog
Aug 3, 2026
Merged

feat(validators): alarm the nominator-counts lane, and size both thresholds to the producer#9311
JSONbored merged 1 commit into
mainfrom
feat/validator-nominator-counts-watchdog

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Two gaps in the alarm coverage over the two outputs of the poller's one Alpha scan.

1. The counts lane had no watchdog

#9302 gave it a D1 sink and metagraphed-infra#243 re-enabled its producer, but nothing watched it — which is the exact shape of the failure that took the lane down in the first place. Its writer targeted a Postgres that went away, nominator_count degraded to null (or to a frozen 2026-08-02 mirror covering 564 of 1,031 validators), and no probe, no red check, no exception fired. The read path degrades so gracefully that the outage was invisible.

nominator_positions — the other output of the same scan — got a watchdog in #9273 for precisely this reason. Its sibling now has the matching one: one MAX() read, a pure rule, a summary rather than a throw, one exception per stale tick under watchdog:validator-nominator-counts-staleness. An empty table alerts too, since that is the state in which every nominator_count is still coming from the frozen mirror or serving null outright.

2. The positions threshold undercut its own producer

NOMINATOR_POSITIONS_STALENESS_THRESHOLD_MS was 6 hours, chosen while that lane had no producer at all, on the reasoning that six hours was "several missed passes at any plausible cadence".

The producer now feeding it runs on a 24h tick (VALIDATOR_NOMINATORS_POLL_SECS defaults to 24 * 3600, and one scan writes both tables). A healthy lane therefore presents an age anywhere in [0h, 24h + scan], so a 6h threshold would have alerted for roughly three quarters of every day.

It hasn't fired yet only because the table is still empty and takes the no_rows branch instead — it would have started the moment the re-enabled producer posted its first batch. An alarm that always fires is one nobody reads, so I fixed it here rather than leaving a known pager-burner behind a change of mine that activates it.

Both thresholds are now 30h: one missed pass plus slack for the scan (~4 min at the measured ~3,100 rows/sec) and cron jitter. Derived from the cadence rather than picked, and stated as such in both headers so the next cadence change has somewhere to land.

Tests

The regression is pinned directly — a capture from the middle of a 24h cycle must be quiet — on both lanes, so nobody re-tightens the threshold under its cadence without a red test. Four positions-suite fixtures that encoded the old 6h assumption were updated rather than deleted.

Cron 19,49 collides with nothing in workers/config.ts and stays off the */5 raw-capture and */15 probe grids. The suite asserts both that uniqueness and that wrangler.jsonc actually declares the trigger — dispatch keys on the literal string, so an undeclared cron is silently dead code.

npm run lint / typecheck / validate / validate:contract-drift   # all clean
npx vitest run tests/validator-nominator-counts-staleness-watchdog.test.ts \
  tests/nominator-positions-staleness-watchdog.test.ts          # 28 passed
npx vitest run tests/api-coverage.test.ts tests/check-worker-deploy-drift.test.ts \
  tests/usage-telemetry.test.ts tests/freshness-watchdog.test.ts \
  tests/lakehouse-seam-watchdog.test.ts                         # 510 passed

Patch coverage verified by diff intersection: 0 uncovered lines, 0 uncovered branches across all four changed source files.

Closes #9310
Part of #9146

…sholds to the producer

Two gaps in the alarm coverage over the two outputs of the poller's one Alpha
scan.

THE COUNTS LANE HAD NO WATCHDOG. #9302 gave it a D1 sink and metagraphed-infra
#243 re-enabled its producer, but nothing watched it -- which is the exact
shape of the failure that produced the lane's outage in the first place. Its
writer targeted a Postgres that went away, nominator_count degraded to null or
to a frozen 2026-08-02 mirror covering 564 of 1,031 validators, and no probe,
no red check and no exception fired. The read path degrades so gracefully that
the outage was invisible. nominator_positions -- the other output of the same
scan -- got a watchdog in #9273 for precisely this reason; its sibling now has
the matching one, same shape as neurons/positions/chain-detail before it: one
MAX() read, a pure rule, a summary rather than a throw, one exception per stale
tick. An EMPTY table alerts too, since that is the state in which every
nominator_count is still coming from the frozen mirror.

THE POSITIONS THRESHOLD UNDERCUT ITS OWN PRODUCER. It was 6 hours, chosen while
that lane had no producer at all, on the reasoning that six hours was "several
missed passes at any plausible cadence". The producer now feeding it runs on a
24h tick (VALIDATOR_NOMINATORS_POLL_SECS defaults to 24*3600, and one scan
writes both tables), so a healthy lane presents an age anywhere in
[0h, 24h+scan] and a 6h threshold would have alerted for roughly three quarters
of every day. It has not fired yet only because the table is still empty and
takes the no_rows branch instead; it would have started the moment the
re-enabled producer posted. An alarm that always fires is one nobody reads.

Both thresholds are now 30h -- one missed pass plus slack for the scan itself
(~4 min at the measured ~3,100 rows/sec) and cron jitter. Derived from the
cadence, not picked, and stated as such in both headers so the next cadence
change has somewhere to land. Tests pin the regression directly: a capture from
the middle of a 24h cycle must be quiet.

Cron 19,49 collides with nothing in workers/config.ts and stays off the */5
raw-capture and */15 probe grids; the suite asserts both that uniqueness and
that wrangler.jsonc actually declares the trigger, since dispatch keys on the
literal string and an undeclared cron is silently dead code.

Closes #9310
Part of #9146
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-registry-sync-api c4b38da Aug 03 2026, 08:55 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-data-api c4b38da Aug 03 2026, 08:55 PM

@superagent-security

Copy link
Copy Markdown

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit 9d34bd4 into main Aug 3, 2026
7 checks passed
@JSONbored
JSONbored deleted the feat/validator-nominator-counts-watchdog branch August 3, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The validator-nominator-counts lane has no watchdog, and the positions watchdog's threshold undercuts its producer's cadence

1 participant