Skip to content

link to https://github.com/kozistr/pytorch_optimizer #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

link to https://github.com/kozistr/pytorch_optimizer #50

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ Net results: 18.18% greater accuracy with Ranger21 vs Adam, same training epochs


### Ranger21 Status:</br>
<b> 2023-05-01 the most recently updated version can be found in [kozistr/pytorch_optimizer](https://github.com/kozistr/pytorch_optimizer)</b>

<b> July 10: 3 new improvements to Ranger21 </b> Three new items have been added to Ranger21 after testing on sub ImageNet benchmark:
1. **Gradient Normalization** - this continues the Gradient centralization concept by normalizing the gradient (vs. gradient centralization subtracts the mean). On ImageNet it produces faster convergence in the first 20 or so epochs.
2. **Softplus transform** - by running the final variance denom through the softplus function, it lifts extremely small values to keep them viable. This helps with refining the training updates and in testing on our sub ImageNet benchmark, it set a new high in accuracy and val loss. (usage: softplus = True is default, set to False at init to turn off). Please see https://arxiv.org/abs/1908.00700 for the original paper.
Expand Down