Skip to content

Commit 704e941

Browse files
authored
Update README.md
1 parent 6fde475 commit 704e941

File tree

1 file changed

+10
-15
lines changed

1 file changed

+10
-15
lines changed

README.md

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,16 @@
1-
# Ranger-Deep-Learning-Optimizer
2-
Ranger - a synergistic optimizer combining RAdam (Rectified Adam) and LookAhead in one codebase.
1+
# Ranger-Deep-Learning-Optimizer
2+
</br>
3+
Ranger - a synergistic optimizer combining RAdam (Rectified Adam) and LookAhead, and now GC (gradient centralization) in one optimizer.
4+
</br>
5+
Latest version 20.4.11 - new record for accuracy on benchmarks with the addition of Gradient Centralization! </br> </br>
6+
What is Gradient Centralization? = "GC can be viewed as a projected gradient descent method with a constrained loss function. The Lipschitzness of the constrained loss function and its gradient is better so that the training process becomes more efficient and stable." Source paper: https://arxiv.org/abs/2004.01461v2
37

4-
Latest version 9.3.19 - full refactoring for slow weights and one pass handling (vs two before). Refactor should eliminate any random save/load issues regarding memory.
8+
Ranger now uses Gradient Centralization by default, and applies it to all conv and fc layers by default. However, everything is customizable so you can test with and without on your own datasets. (Turn on off via "use_gc" flag at init).
59

10+
# Full customization at init:
611

7-
///////////////////////
8-
9-
Beta Version - Ranger913A.py:
10-
11-
For anyone who wants to try this out early, this version changes from RAdam to using calibrated anistropic adaptive learning rate per this paper:
12-
13-
https://arxiv.org/abs/1908.00700v2
14-
15-
"Empirical studies support our observation of the anisotropic A-LR and show that the proposed methods outperform existing AGMs and generalize even better than S-Momentum in multiple deep learning tasks."
16-
17-
Initial testing looks very good for training stabilization. Any feedback in comparsion with current Ranger (9.3.19) is welcome!
18-
12+
13+
![](images/ranger-init.jpg)
1914
/////////////////////
2015

2116
Medium article with more info:

0 commit comments

Comments
 (0)