Add scheduler

[rentainhe]

TODO

Add more scheduler

• PlateauLRScheduler ( ReduceLROnPlateau 传参 need to fix )
• PolyLRScheduler
• TanhLRScheduler

OneFlow

• ReduceLROnPlateau didn't have epoch args

[rentainhe]

flow.optim.lr_scheduler.ReduceLROnPlateau的step函数超参上缺少了一个epoch参数，可以对齐一下torch里的这个传参

[Ldpe2G]

oeneflow 仓库的 ReduceLROnPlateau 也要改一下，还是先改了 这个再加吧，不然会报错

[Ldpe2G]

一个功能加了要保证能运行

[rentainhe]

Draw LRScheduler Picture

# from timm.scheduler.cosine_lr import CosineLRScheduler
from oneflow import optim
from flowvision.scheduler import MultiStepLRScheduler
from flowvision.models import ModelCreator
import oneflow as flow

model = ModelCreator.create_model("alexnet")
optimizer = flow.optim.SGD(model.parameters())

from matplotlib import pyplot as plt

from typing import List
a = [30, 60, 90]
print([_ * 100 for _ in a])
print(isinstance(a, list))


def get_lr_per_epoch(scheduler, num_epoch):
    lr_per_epoch = []
    for epoch in range(num_epoch):
        lr_per_epoch.append(scheduler.get_epoch_values(epoch))
    return lr_per_epoch

num_epoch = 100
scheduler = MultiStepLRScheduler(optimizer, decay_t=[30, 60, 90], decay_rate=0.1, warmup_lr_init=0.00001, warmup_t=10)

lr_per_epoch = []
for i in range(num_epoch):
    scheduler.step(i)
    lr_per_epoch.append(optimizer.param_groups[0]["lr"])


plt.plot([i for i in range(num_epoch)], lr_per_epoch)
plt.savefig("./test_multi_step.jpg")