Give the HG002 pair fit the control arm it was missing - #739
Merged
Merged
Conversation
Job 22226933 produced the numbers #695 asked for and they look like a result: simulated collapsed-tail rates of 11.95% against a real 11.12% on R1 and 26.36% against 26.22% on R2, where #694 records eidolon at ~27x short. The run cannot support that reading, because it has no control. `validate_quality_degradation.sh` says why in its own header -- "the control arm matters as much as the test arm; without it, a number from the test arm has nothing to sit against" -- and this script carried over its measurement without carrying over that. The available counterfactual was #694's historical 0.36%, measured on a different subsample by an older build, which differs from the test arm in more ways than the one under test. So the same pair is now also fitted with ONE population per mate, generated from, and measured alongside. Six rows instead of four. The two-population row means something only in the gap between the real row and the control row. Smoked on the H1N1 mate pair, all stages, rc=0. There the control matches the target too, which is precisely the "then it is not a result" case the new guidance text describes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Job 22226933 produced the numbers #695 asked for, and they look like a result — simulated collapsed-tail rates of 11.95% against a real 11.12% on R1 and 26.36% against 26.22% on R2, where #694 records eidolon at ~27x short.
The run cannot support that reading, because it has no control.
validate_quality_degradation.shsays why in its own header: "The control arm matters as much as the test arm — without it, a number from the test arm has nothing to sit against." This script carried over its measurement and not that. The only counterfactual available was #694's historical 0.36%, measured on a different subsample by an older build — which differs from the test arm in more ways than the one under test.What changes
The same pair is now also fitted with one population per mate, generated from, and measured alongside. Six rows instead of four:
The closing guidance now says how to read them: the two-pop row means something only in the gap between the real row and the control row. Matching the target is a result if the control falls far short, and is not one if the control matches too.
The subsample is cached in
$WORK, so a rerun skips the expensive pass over the staged FASTQs and only refits.Verified
Smoked on the H1N1 mate pair through the stager's layout, all stages,
rc=0. On that fixture the control matches the target as well — precisely the "then it is not a result" case the new text describes, which is a useful demonstration that the framing discriminates. H1N1 numbers are not evidence about HG002 and none are quoted.🤖 Generated with Claude Code