update with new records, new fixes

Author: lessw2020

README.md

@@ -5,12 +5,15 @@ Medium article with more info:
 https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d
 
 Multiple updates:
+1 - Ranger is the optimizer we used to beat the high scores for 8 different categories on the FastAI leaderboards!  (Previous records all held with AdamW optimizer).
 
-1 - We used Ranger to beat the FastAI leaderboard score by nearly 20% (19.77%).  The trick was to combine Ranger with: Mish activation function, and flat+ cosine anneal training curve.
+2 - Highly recommend combining Ranger with: Mish activation function, and flat+ cosine anneal training curve.
 
-2 - Based on that, also found .95 is better than .90 for beta1 (momentum) param (ala betas=(0.95, 0.999)).
+3 - Based on that, also found .95 is better than .90 for beta1 (momentum) param (ala betas=(0.95, 0.999)).
 
-3 - Verified no load/save issues in our codebase here.  It was an issue for people that were using LookAhead/RAdam as seperate components.
+Fixes:
+1 - Differential Group learning rates now supported.  This was fix in RAdam and ported here thanks to @sholderbach.
+2 - In progress fix - save and then load may leave first run weights stranded in memory, slowing down future runs...trying to investigate and fix now.
 
 
 Usage and notebook to test are available here: