[Question] Step 3 loss curve

[fanminshi]

Question

I performed the step 3 moe-finetunning on phi-2 model, the loss doesn't seemed to drop that much. I wonder if that's normal. Thanks!

[EmberNoWeither]

Hello! When i use the script 'finetune_moe.sh' to train phi-2, i got the similar curves! But I found a problem that my training process will be OOM! It happened randomly and the gpu memory-usage is not constant. The gpu memory-usage will vary widely leading to OOM... Did you find this problem?