Allow an LR less than 1e-4 for Lora.

[JuanForge]

Hello,
I'm trying to train a LoRA with a lower learning rate to improve training quality, but SimpleTuner seems to ignore my configuration.

Even when I set a value like 5e-5, the backend still trains at 1e-4.

Everything looks correct in the intermediate configs, but in the UI I can clearly see the Learning Rate displayed as 1.00e-4.

Is there something overriding the value, or am I missing a parameter?

Latest version in the 'Release' branch (v4.1.2) for a LoRa Z Image

[JuanForge]

Additionally, I’m experiencing instability with the step loss during training.

The loss fluctuates significantly, which often results in newer checkpoints performing worse than earlier ones.

This makes it difficult to identify the best model without manually testing every version.

Do you have any recommendations to stabilize the training and reduce these fluctuations?

Thanks in advance.

[bghira]

you're likely resuming training and trying to change the learning rate. but the optimiser state resumes with the learning rate from the previous run. there's nothing actually preventing (in the codebase) a lower learning rate from being used. i think it's just the resume blocking it.

[JuanForge]

Hi,
I’ve done some testing and confirmed that the learning rate wasn’t updating because I resumed training from a checkpoint. Thanks for pointing that out.

Do you have any recommendations to stabilize the step loss? Right now it decreases but remains quite noisy, which forces me to manually test multiple checkpoints to find the best one.

For reference, here are some recent step loss values:
0.428, 0.269, 0.266, 0.298, 0.145, 0.328, 0.362, 0.427, 0.343, 0.331, 0.253, 0.569, ...
This instability is also noticeable during inference.

My current setup:

Batch size: 1
Gradient accumulation: 1 (also tested 4, 8, 12, 32 without significant improvement)
Learning rate: 9e-5 (constant)
Max gradient norm: 2.0

Sorry for the late reply.

( translated by: GPT )