Loss scaling is incorrect when using gradient_accumulation_steps > 1

[BitPhinix]

🐛 Describe the bug

Fixed it for this one specific case here, sadly don't have the time to put in a proper pr for all models for all cases: BitPhinix@c455526

TLDR:

Ever since huggingface/transformers#34191

Transformers expect all models that take in kwargs to scale the loss by num_items_in_batch.

Right now, the loss is effectively multiplied by the number of gradient accumulation steps steps when using liger kernels

Reproduce

No response

Versions

all

[Tcc0403]

AFAIK, most hugginface models with gradient accumulation fix are calling self.loss_function(logits, labels, self.vocab_size, **kwargs) which is fixed_cross_entropy() under the hood. Liger has a similar function defined as LigerForCausalLMLoss().

Any model that is originally calling self.loss_function(logits, labels, self.vocab_size, **kwargs) should have LigerForCausalLMLoss() patched already. I might overlook so feel free to tell me if there wasn't the case.

Regarding gemma3 multimodal or some specific models, in huggingface implementation, there's also a PR about it recently. huggingface/transformers#37208

In gemma3 multidomal forward(), it doesn't take num_items_in_batch and scale the loss either. That's why I didn't add loss scaling to it as it would differ from huggface's training result.