validator: add --validator.cycle_offset_minutes for wall-clock lane scheduling#297
Merged
Conversation
…cheduling When set, the sequential scheduler anchors prompt cycles to minute boundaries where minutes-since-epoch % cycle_interval_minutes equals the offset, instead of spacing cycles relative to the previous one. This lets several validator processes cover the same asset on interleaved lanes, and an overrunning cycle re-locks to its own lane instead of drifting. Unset keeps the existing relative schedule. Only applies to the prompt cycles; scoring ignores it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an optional wall-clock “lane” mode for sequential prompt scheduling so multiple validator processes can interleave prompt cycles on minute boundaries without drifting, while preserving the current relative schedule by default.
Changes:
- Introduces
PromptConfig.cycle_offset_minutesand wires a new--validator.cycle_offset_minutesCLI flag to apply it to the active (low/high) cycle config. - Implements wall-clock lane selection in
SequentialScheduler.select_grid_delay, with overrun detection/warnings. - Adds unit tests covering lane interleaving, steady-state lane adherence, overrun re-locking, and default-path regression.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sequential_scheduler.py | Adds unit tests for wall-clock lane scheduling behavior and regression coverage for the default path. |
| synth/validator/prompt_config.py | Adds cycle_offset_minutes to the prompt configuration model (default None). |
| synth/utils/sequential_scheduler.py | Adds grid/lane-based delay computation when cycle_offset_minutes is set. |
| synth/utils/config.py | Adds the new --validator.cycle_offset_minutes CLI flag. |
| neurons/validator.py | Merges CLI asset filtering and new lane offset override into _apply_cycle_overrides, including range validation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+102
to
+109
| interval = prompt_config.cycle_interval_minutes | ||
|
|
||
| now = get_current_time() | ||
| next_boundary = round_time_to_minutes(now) | ||
| boundary_minutes = int(next_boundary.timestamp()) // 60 | ||
| next_cycle = next_boundary + timedelta( | ||
| minutes=(offset - boundary_minutes) % interval | ||
| ) |
Comment on lines
+226
to
+230
| parser.add_argument( | ||
| "--validator.cycle_offset_minutes", | ||
| type=int, | ||
| default=None, | ||
| ) |
Two unprotected chain-sync paths could kill the process when the public endpoint rate-limits the heavy metagraph runtime call per IP (websocket handshake rejected with HTTP 429): - BaseNeuron.__init__'s first metagraph fetch — now retried with jittered exponential backoff (tenacity, same pattern as save_responses), so a boot-time burst is waited out in-process instead of crash-looping, which kept re-triggering the limit. - Validator.refresh_metagraph — a raising sync is now swallowed: the previous miner set stays in use and the unstamped timer makes the next cycle retry. This also protects the scoring loop, which calls it outside any try/except. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an optional
--validator.cycle_offset_minutesflag (default: unset, no behavior change). When set, the sequential scheduler anchors the prompt cycle to the wall clock: prompts fire on minute boundaries whereminutes_since_epoch % cycle_interval_minutes == offset, instead of being spaced relative to the previous cycle.Why
A single prompt process cannot go below a ~2-minute effective cadence: each cycle occupies most of its minute waiting for miner responses. With wall-clock lanes, several validator processes can cover the same asset on interleaved offsets (e.g. interval 2 with offsets 0 and 1 combine to one prompt per minute), each keeping the full miner-response window. Anchoring to the wall clock also means an overrunning cycle skips to the next slot in its own lane instead of drifting into a sibling's.
Details
PromptConfig.cycle_offset_minutes(int | None) — None keeps the existing relative schedule for all current deployments._apply_cycle_overrides(merged with the assets filter) applies the flag to the active cycle's config and validates the offset is in[0, cycle_interval_minutes); the scoring cycle ignores it.SequentialScheduler.select_grid_delayimplements the grid math; a missed slot logs a warning and re-locks to the same lane.🤖 Generated with Claude Code