Fix(validator): HF Hub flake in foreground baseline must not kill the process - #192
Open
present42 wants to merge 1 commit into
Open
Fix(validator): HF Hub flake in foreground baseline must not kill the process#192present42 wants to merge 1 commit into
present42 wants to merge 1 commit into
Conversation
…he process 2026-07-10 10:30 UTC, round 8590274: a transient `requests.ConnectionError (huggingface.co Read timed out)` inside the foreground baseline dataloader build (`_evaluate_on_fresh_loader_sync` → `load_streaming_shard` → `datasets.load_dataset`) propagated uncaught through `evaluate_foreground_round` and `_bounded_foreground_eval` into run()'s top-level "Quit training" handler — killing the validator. Cost: the whole round finalized scored_count=0 and the cycle fell back to even weights. Two layers, mirroring the background worker's existing policy (DATALOADER_BUILD_RETRY_DELAYS_SEC), which already survives these blips: - evaluator.py: retry the baseline build with backoff (0/10/30s); on exhaustion, skip the foreground pass for the round and return cleanly — bg-eval still covers the roster and finalize proceeds. CancelledError is re-raised so the outer validate-phase deadline keeps working. - run.py: catch non-timeout exceptions at the foreground call site and continue the round with partial scores — foreground eval is best-effort; nothing downstream requires it to have succeeded. Safety net for any future non-HF failure in the same path. Tests pin the degraded path (returns [] / the completed_out alias instead of raising), retry-then-recover, and no spurious retries on success. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: HF Hub flake in foreground baseline was process-fatal.
On 2026-07-10 (round 8590274) a transient huggingface.co read timeout inside the foreground baseline dataloader build escaped every handler and killed the validator — the round finalized with 0 scores and the cycle degraded to fallback weights. The background worker already survives these blips via bounded retry + degrade; this PR gives the foreground path the same policy (retry 0/10/30s, then skip the round's foreground pass instead of raising) plus a catch-and-continue at the run.py call site as a safety net for any other failure in that path. Foreground eval is best-effort by design — bg-eval covers the roster and finalize never required it. Tests pin the degraded path, the completed_out aliasing contract, and retry-recovery. (The 4 failing suite tests are pre-existing on staging.)