[Badcase]: Loss does not drop when using Liger Kernel at Qwen2.5

[Se-Hun]

Has this been raised before?

• I have checked the GitHub README.
• I have checked the Qwen documentation and cannot find an answer there.
• I have searched the issues and there is not a similar one.
• I confirm that this is not a bug report, a feature request, or a badcase.

Description

Hello,

In my case:

I am trying to instruction tuning Qwen2.5-14B-Instruct with Liger Kernel.

There is my question:

I know that the liger kernel is supported in the dev version of huggingface transformers. However, when training the Qwen2.5 model with Liger Kernel, the loss value does not drop. Not supported yet at Qwen2.5?

[jklj077]

Hi, unfortunately, I don't think we have any experience with it. Would mind reporting this to the Liger Kernel maintainers?

[jklj077]

Hi, could you please provide more info on this? did you use frameworks like axolotl, llama-factory, or others? which type of chat templates and which script did you run? transformers have released a version that support Liger Kernel, could you give it a try?