Set RNG seed manually with '-rg' parameter #654
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I tried to fix this PR properly but it gets really tricky with the seeds because there are multiple of them (for stochastic rounding vs. for init), of different dtypes and it feels kind of delicate. Might revisit later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a '-rg' parameter to manually set the RNG seed. This is useful to see if a change is beneficial or not when the difference is potentially real but smaller than the noise threshold. As we are fully deterministic (for a given system), we can't just run multiple time with the same settings anymore.
The same seed will be used for both gpt2_init_common() and gpt2_build_from_random(), so this effectively changes the default seed from 42 to 13371337 for gpt2_build_from_random(). It does not change the default seed for the non-random initialisation path or test_gpt2cu in any way.