🔄 dataset: revert subnet to Nemotron-CC-Math + C4 (exp_nemotron_c4, group 4) - #203
Merged
Merged
Conversation
…chema v3)
Startup recovery restored telemetry at most ONCE, and `val_loss` had no
recovery path at all. Two Watchtower restarts 25 minutes apart on
2026-07-31 left every per-miner family at zero series for 17 minutes and
lost cycle 16684's losses permanently.
Three coupled gaps, one schema bump:
1. `val_loss` is journaled (v3 `uid_to_val_loss`). It is published at eval
time and is NOT derivable from `scores`: `delta = max(0.0, baseline -
val_loss)` clamps at zero, so every miner scoring 0 — the majority in
many rounds — would be underivable. Plumbed through `MinerEvalJob.val_loss`
→ `Round.mark_scored(..., val_loss=)` → `Round.val_losses` → journal;
both eval paths (foreground + background) pass it.
2. Recovery now re-emits telemetry on EVERY restart, not just the first.
The replay loop only touches unfinalized journals, and replaying one
marks it finalized — so the second restart found nothing and emitted
nothing. New `republish_telemetry_from_journal` reads the newest
finalized journal and sets the gauges directly.
It is deliberately METRICS-ONLY. Re-running `finalize_round_scores`
would keep the aggregator's point set correct (`drop_round` runs first)
but re-stamp those points with fresh `_utc_now()` timestamps, shoving
them to the end of the time-ordered series — and the rolling average is
"last N by timestamp", which is what drives weight submission. A
re-finalize would therefore silently reshuffle the scoring window. A
test asserts the pass makes zero mutating aggregator calls.
3. The round-level counters (`validator_round_miners_{scored,pending,
failed}`, `lifecycle_step`, `current_round_id`) now restore via that
same pass, so they cover the finalized path too rather than inheriting
the once-only limitation. `roster_size` + `lifecycle_step` are journaled
(v3) to feed them; pre-v3 journals clamp pending to 0 rather than
inventing a denominator.
`verdict_uids` mirrors finalize's entry set exactly — scored ∪
validation-failed ∪ freeze-zero, excluding operational failures, which get
no aggregator entry so telemetry must not imply they were judged.
`from_json` still accepts v1/v2 (missing fields default). Verified against
the v0.3.4 image: the suite's failure set is byte-identical before and
after this change (4 pre-existing failures, unrelated), and the 49 tests
across the journal/telemetry suites pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…roup 4) PR #186 switched the subnet from math data to legal data. Despite touching 9 files, the switch itself was one line: the locked `TaskCfg.expert_group_name` default. This reverts the DATASET while keeping everything layered on top since — the anti-memorization eval pipeline, the telemetry/recovery work, and the bittensor 10.5 chain fix all live in shared code and are dataset-agnostic. New expert group `exp_nemotron_c4` (group_id 4) rather than reactivating exp_math (group 0), so the cutover is clean and doesn't inherit group 0's history. `exp_math` and `exp_legal` are left on disk for reference. Two improvements are retained deliberately rather than by accident: - `dataset_class` is intentionally UNSET, which resolves to `DefaultStreamingTorchDataset` and therefore `tokenize_windowed`. `exp_math/dataset.py` still prefix-truncates, so pointing at it would have silently dropped the windowing fix ("exp_math left for follow-up", d7e979c). - Both sources are pinned to commit SHAs via `eval_source_revision_pin`. `_KNOWN_SOURCES` declares `revision="main"` for c4/Nemotron, and `main` is a moving target — an HF re-upload mid-rollout would make two validators read different rows for the same seed and break weight consensus. The legal-tuned eval gates are inherited unchanged after measuring them against the new mix (600 rows/source): c4 0% empty / 5.5% under 200 chars, Nemotron 0% / 0%, prefix-dedup 0% on both — versus Multi_Legal_Pile's 38% empty and 75% duplicate prefixes. Effectively inert here, so no override. Expert assignment is copied from exp_math, verified valid for the current post-#188 (2Fnat) architecture rather than assumed: #188 regenerated that file itself (68a878e, 208->187 experts) and nothing architectural has landed since. Confirmed by building the partial model — 52 MoE modules across layers 1..26 for trainable group 4 + helper group 2, n_routed_experts=64, expert ids 0..63 all in range. Docs: miner-faq updated for the new group, plus three stale claims corrected while in there — "8 experts per MoE layer" (it is 64), "miners self-select their group" (the field is locked and resets on load), and `sequence_length=4096` (1024 since #188). exp_legal README and the migration plan are marked historical. ROLLOUT — fleet-wide flag day, not a rolling change: - Validators only score miners whose chain commit `expert_group` matches, so miners still on group 3 become invisible and earn nothing until they upgrade. Announce before tagging. - `nvidia/Nemotron-CC-Math-v1` is GATED. Every validator and miner HF_TOKEN must belong to an account that accepted the license — metadata reads fine without it, so this fails late at dataloader build with GatedRepoError, not at startup. - Fresh checkpoint dir (checkpoint_path is group-scoped), so the model restarts from base and baseline loss re-converges over several cycles. Verified in the v0.3.4 image: config resolution, locked-field reset from a stale exp_legal YAML, seeded shard-pick on both sources honouring the pins, non-prefix windowing, live eval stream, and the full suite showing exactly the 4 pre-existing master failures and no new ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both changes require a coordinated fleet restart, so they ship in one release rather than two. They are independent — disjoint file sets, common base 3425d0d — but combining them means miners and validators absorb a single restart instead of two, which matters here because a restart abandons the in-flight background evaluation. Merging this branch will auto-close PR #200; its commit rides along unchanged.
isabella618033
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #186 switched the subnet from math data to legal data. Despite touching 9 files, the switch itself was one line: the locked
TaskCfg.expert_group_namedefault. This reverts the DATASET while keeping everything layered on top since — the anti-memorization eval pipeline, the telemetry/recovery work, and the bittensor 10.5 chain fix all live in shared code and are dataset-agnostic.New expert group
exp_nemotron_c4(group_id 4) rather than reactivating exp_math (group 0), so the cutover is clean and doesn't inherit group 0's history.exp_mathandexp_legalare left on disk for reference.Two improvements are retained deliberately rather than by accident:
dataset_classis intentionally UNSET, which resolves toDefaultStreamingTorchDatasetand thereforetokenize_windowed.exp_math/dataset.pystill prefix-truncates, so pointing at it would have silently dropped the windowing fix ("exp_math left for follow-up", d7e979c).eval_source_revision_pin._KNOWN_SOURCESdeclaresrevision="main"for c4/Nemotron, andmainis a moving target — an HF re-upload mid-rollout would make two validators read different rows for the same seed and break weight consensus.The legal-tuned eval gates are inherited unchanged after measuring them against the new mix (600 rows/source): c4 0% empty / 5.5% under 200 chars, Nemotron 0% / 0%, prefix-dedup 0% on both — versus Multi_Legal_Pile's 38% empty and 75% duplicate prefixes. Effectively inert here, so no override.
Expert assignment is copied from exp_math, verified valid for the current post-#188 (2Fnat) architecture rather than assumed: #188 regenerated that file itself (68a878e, 208->187 experts) and nothing architectural has landed since. Confirmed by building the partial model — 52 MoE modules across layers 1..26 for trainable group 4 + helper group 2, n_routed_experts=64, expert ids 0..63 all in range.
Docs: miner-faq updated for the new group, plus three stale claims corrected while in there — "8 experts per MoE layer" (it is 64), "miners self-select their group" (the field is locked and resets on load), and
sequence_length=4096(1024 since #188). exp_legal README and the migration plan are marked historical.ROLLOUT — fleet-wide flag day, not a rolling change:
expert_groupmatches, so miners still on group 3 become invisible and earn nothing until they upgrade. Announce before tagging.nvidia/Nemotron-CC-Math-v1is GATED. Every validator and miner HF_TOKEN must belong to an account that accepted the license — metadata reads fine without it, so this fails late at dataloader build with GatedRepoError, not at startup.Verified in the v0.3.4 image: config resolution, locked-field reset from a stale exp_legal YAML, seeded shard-pick on both sources honouring the pins, non-prefix windowing, live eval stream, and the full suite showing exactly the 4 pre-existing master failures and no new ones.