validator: survive chain-endpoint rate limiting instead of crashing#298
Closed
Thykof wants to merge 2 commits into
Closed
validator: survive chain-endpoint rate limiting instead of crashing#298Thykof wants to merge 2 commits into
Thykof wants to merge 2 commits into
Conversation
Two unprotected chain-sync paths could kill the process when the public endpoint rate-limits the heavy metagraph runtime call per IP (websocket handshake rejected with HTTP 429): - BaseNeuron.__init__'s first metagraph fetch — now retried with jittered exponential backoff (tenacity, same pattern as save_responses), so a boot-time burst is waited out in-process instead of crash-looping, which kept re-triggering the limit. - Validator.refresh_metagraph — a raising sync is now swallowed: the previous miner set stays in use and the unstamped timer makes the next cycle retry. This also protects the scoring loop, which calls it outside any try/except. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thykof
commented
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The public chain endpoints rate-limit the heavy metagraph runtime call per source IP — the websocket handshake itself is rejected with HTTP 429, which surfaces as
InvalidStatus/MaxRetriesExceeded. Two sync paths had no protection against this, and both kill the process:BaseNeuron.__init__— the firstsubtensor.metagraph(netuid)fetch. When several neurons (re)start at once behind one IP, the unretried call crashes the process; the synchronized container restarts then keep re-triggering the limit (restart loops skip any startup stagger). Now retried with jittered exponential backoff (tenacity,reraise=True, up to 10 attempts / 120 s max wait — same pattern assave_responses), so the burst is waited out in-process and callers desynchronize naturally.Validator.refresh_metagraph— the boot call inforward_validatorand the scoring loop's call run outside the scheduler'stry/except. A raising sync is now swallowed with a logged exception: the previous miner set stays in use, and since the timer isn't stamped, the next cycle retries. This mirrors the method's existing degenerate-empty-sync semantics.Observed in production this weekend: a namespace-wide rollout put most validator pods into CrashLoopBackOff on boot 429s, while pods that got past init handled the same 429 gracefully mid-cycle.
Tests
tests/test_refresh_metagraph.py: a raising sync neither propagates nor stamps the retry timer and keeps the previous miner set; a successful sync stamps it.🤖 Generated with Claude Code