perf(validator): schedule metagraph sync and metagraph_history snapshots#296
Open
Thykof wants to merge 1 commit into
Open
perf(validator): schedule metagraph sync and metagraph_history snapshots#296Thykof wants to merge 1 commit into
Thykof wants to merge 1 commit into
Conversation
metagraph_history gained a full snapshot every prompt cycle because the write was coupled to the cycle cadence. Decouple it: prompt processes refresh the chain metagraph (miner_uids, axons, miner identities) at most every 15 minutes with no snapshot; the scoring process is the only snapshot writer, at most every 30 minutes, including at boot. Miner identity rows are now upserted on every refresh so a newly queryable uid always has its miners row before its first responses are saved. An empty (degenerate) sync result is applied but not stamped, so the next cycle retries; the refresh runs after forward_prompt so the chain sync never delays a minute-aligned prompt slot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR decouples metagraph syncing and metagraph_history snapshotting from the prompt cadence by introducing a dedicated refresh scheduler and a helper gate function, reducing unnecessary snapshot volume while keeping miner identity/address data fresh for querying.
Changes:
- Added
metagraph_refresh_due()helper (with unit tests) to gate refresh operations by interval. - Updated validator cycle loop(s) to refresh the metagraph on a separate schedule, writing
metagraph_historysnapshots only from the scoring process. - Adjusted miner identity upserts so miner records are refreshed independently of snapshot writes, and renamed the forward helper parameter to
save_snapshot.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
tests/test_helpers.py |
Adds unit coverage for the new metagraph_refresh_due() interval gate. |
synth/validator/forward.py |
Renames save to required save_snapshot, always upserts miner identities, and gates metagraph_history writes by caller schedule. |
synth/utils/helpers.py |
Introduces metagraph_refresh_due() helper for refresh scheduling decisions. |
neurons/validator.py |
Implements per-cycle metagraph refresh scheduling and ensures only the scoring cycle writes snapshots. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+209
to
+213
| An empty result (degenerate chain response) is applied — the shared | ||
| metagraph object was just emptied by the sync, so stale uids must | ||
| not outlive it — but not stamped, so the next cycle retries. | ||
| """ | ||
| now = get_current_time() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
metagraph_historyreceives a full snapshot (one row per available uid) on every low-frequency prompt cycle, because the write is coupled to the prompt cadence. At short cycle intervals that is a large volume of near-identical rows per day with no consumer that needs that granularity — downstream analytics read the table on an hourly basis.Change
Give the metagraph work its own schedule, decoupled from prompt cadence (constants in
neurons/validator.py; each cycle runs as its own process with its own timer):miner_uids, axon addresses, and miner identity rows — at most everyMETAGRAPH_SYNC_INTERVAL_MINUTES = 15, writing no snapshot. The refresh runs afterforward_prompt, so the chain sync never delays a minute-aligned prompt slot.metagraph_historywriter, at most everyMETAGRAPH_SNAPSHOT_INTERVAL_MINUTES = 30(including the boot snapshot; prompt-process restarts append nothing).insert_new_minersnow runs on every refresh (the on-conflictupdated_atbump is kept — it makes a re-current identity the newest row for its uid). Sinceminer_uidsand the identity rows travel through the same call, a newly queryable uid always has itsminersrow before its first responses are saved.forward_promptskips cleanly) but the timer is not stamped, so the next cycle retries instead of waiting a full interval.metagraph_refresh_dueinsynth/utils/helpers.pywith unit tests; thesaveparameter is renamed tosave_snapshot(identities are no longer gated by it) and made required.Verification
black --check/flake8clean; newtest_metagraph_refresh_duecovers the gate boundaries (never-refreshed, below/at/above interval).🤖 Generated with Claude Code