📊 telemetry: publish round progress from the foreground path too - #201
Merged
george-connito merged 1 commit intoJul 31, 2026
Merged
Conversation
`validator_round_miners_{scored,failed,pending}` were written only by
`BackgroundEvalWorker._record_metrics`, whose loop starts when the eval
window opens at Merge. Foreground evals run earlier, during Submission,
and accumulate in `scored_uids` with nothing publishing them — so the
dashboard's "Evaluated N of M" panel showed the previous round's final
count for the first ~14 minutes of every round, then jumped straight to
the foreground total (measured on cycle 16674: Submission 21:32, panel
flipped at Merge 21:46 with a first value of 4).
The counts were always correct — `evaluate_foreground_round` does call
`mark_scored`, and `Round.stats()` reads whole-round state. Only the
publish was background-scoped.
Lift the publish out of `BackgroundEvalWorker` into
`telemetry.set_round_progress(round_id, scored, failed, pending)` +
`Round.publish_progress()`, and call it from:
- `Round.freeze` (via run.py, next to the lifecycle-step-0 publish) so
the round appears in the metric at freeze with scored=0;
- each foreground `mark_scored` / `mark_failed` /
`mark_validation_failed`, matching the background worker's existing
mark → publish → prune ordering;
- the background worker, now via the shared method.
`note_round_series` moves into the shared helper, so the labelsets are
still evicted on the normal cutoff. The vestigial `scored_inc` argument
(never read) is dropped. The download worker's partial pending/failed
publishes are left untouched — out of scope and still correct.
Tests: new `test_round_progress_publish.py` covers the shared helper
(all three gauges, eviction registration, never-raises), freeze-time
seeding, the foreground regression (mark_scored → counter moves with no
background worker), failure/validation-failure counting, stats parity,
lock-safety, and per-round isolation. Full round/telemetry suite (87)
green in the stable image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
isabella618033
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
validator_round_miners_{scored,failed,pending}were written only byBackgroundEvalWorker._record_metrics, whose loop starts when the eval window opens at Merge. Foreground evals run earlier, during Submission, and accumulate inscored_uidswith nothing publishing them — so the dashboard's "Evaluated N of M" panel showed the previous round's final count for the first ~14 minutes of every round, then jumped straight to the foreground total (measured on cycle 16674: Submission 21:32, panel flipped at Merge 21:46 with a first value of 4).The counts were always correct —
evaluate_foreground_rounddoes callmark_scored, andRound.stats()reads whole-round state. Only the publish was background-scoped.Lift the publish out of
BackgroundEvalWorkerintotelemetry.set_round_progress(round_id, scored, failed, pending)+Round.publish_progress(), and call it from:Round.freeze(via run.py, next to the lifecycle-step-0 publish) so the round appears in the metric at freeze with scored=0;mark_scored/mark_failed/mark_validation_failed, matching the background worker's existing mark → publish → prune ordering;note_round_seriesmoves into the shared helper, so the labelsets are still evicted on the normal cutoff. The vestigialscored_incargument (never read) is dropped. The download worker's partial pending/failed publishes are left untouched — out of scope and still correct.Tests: new
test_round_progress_publish.pycovers the shared helper (all three gauges, eviction registration, never-raises), freeze-time seeding, the foreground regression (mark_scored → counter moves with no background worker), failure/validation-failure counting, stats parity, lock-safety, and per-round isolation. Full round/telemetry suite (87) green in the stable image.