Ranger - a synergistic optimizer combining RAdam (Rectified Adam) and LookAhead in one codebase. (I built on LonePatient's implementation for the Lookahead part, and the official RAdam code for RAdam).
Usage and notebook to test with comming shortly.