Skip to content

📊 telemetry: publish round progress from the foreground path too - #201

Merged
george-connito merged 1 commit into
masterfrom
feat/publish-round-progress-from-foreground
Jul 31, 2026
Merged

📊 telemetry: publish round progress from the foreground path too#201
george-connito merged 1 commit into
masterfrom
feat/publish-round-progress-from-foreground

Conversation

@george-connito

Copy link
Copy Markdown
Collaborator

validator_round_miners_{scored,failed,pending} were written only by BackgroundEvalWorker._record_metrics, whose loop starts when the eval window opens at Merge. Foreground evals run earlier, during Submission, and accumulate in scored_uids with nothing publishing them — so the dashboard's "Evaluated N of M" panel showed the previous round's final count for the first ~14 minutes of every round, then jumped straight to the foreground total (measured on cycle 16674: Submission 21:32, panel flipped at Merge 21:46 with a first value of 4).

The counts were always correct — evaluate_foreground_round does call mark_scored, and Round.stats() reads whole-round state. Only the publish was background-scoped.

Lift the publish out of BackgroundEvalWorker into telemetry.set_round_progress(round_id, scored, failed, pending) + Round.publish_progress(), and call it from:

  • Round.freeze (via run.py, next to the lifecycle-step-0 publish) so the round appears in the metric at freeze with scored=0;
  • each foreground mark_scored / mark_failed / mark_validation_failed, matching the background worker's existing mark → publish → prune ordering;
  • the background worker, now via the shared method.

note_round_series moves into the shared helper, so the labelsets are still evicted on the normal cutoff. The vestigial scored_inc argument (never read) is dropped. The download worker's partial pending/failed publishes are left untouched — out of scope and still correct.

Tests: new test_round_progress_publish.py covers the shared helper (all three gauges, eviction registration, never-raises), freeze-time seeding, the foreground regression (mark_scored → counter moves with no background worker), failure/validation-failure counting, stats parity, lock-safety, and per-round isolation. Full round/telemetry suite (87) green in the stable image.

`validator_round_miners_{scored,failed,pending}` were written only by
`BackgroundEvalWorker._record_metrics`, whose loop starts when the eval
window opens at Merge. Foreground evals run earlier, during Submission,
and accumulate in `scored_uids` with nothing publishing them — so the
dashboard's "Evaluated N of M" panel showed the previous round's final
count for the first ~14 minutes of every round, then jumped straight to
the foreground total (measured on cycle 16674: Submission 21:32, panel
flipped at Merge 21:46 with a first value of 4).

The counts were always correct — `evaluate_foreground_round` does call
`mark_scored`, and `Round.stats()` reads whole-round state. Only the
publish was background-scoped.

Lift the publish out of `BackgroundEvalWorker` into
`telemetry.set_round_progress(round_id, scored, failed, pending)` +
`Round.publish_progress()`, and call it from:
  - `Round.freeze` (via run.py, next to the lifecycle-step-0 publish) so
    the round appears in the metric at freeze with scored=0;
  - each foreground `mark_scored` / `mark_failed` /
    `mark_validation_failed`, matching the background worker's existing
    mark → publish → prune ordering;
  - the background worker, now via the shared method.

`note_round_series` moves into the shared helper, so the labelsets are
still evicted on the normal cutoff. The vestigial `scored_inc` argument
(never read) is dropped. The download worker's partial pending/failed
publishes are left untouched — out of scope and still correct.

Tests: new `test_round_progress_publish.py` covers the shared helper
(all three gauges, eviction registration, never-raises), freeze-time
seeding, the foreground regression (mark_scored → counter moves with no
background worker), failure/validation-failure counting, stats parity,
lock-safety, and per-round isolation. Full round/telemetry suite (87)
green in the stable image.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@george-connito
george-connito merged commit 4a73fd0 into master Jul 31, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants