The training is unstable

[shengkelong]

Thank you for your impressive work. But when I try to recurrent this network(I rewrite the code myself), sometimes the loss will suddenly increase by 10 times. The structure of the network is correct because I can load the pretrained network, so I think there may be some details I didn't notice. Could you tell me what methods you have taken in training to ensure stability?

[pkuxmq]

We restrict the range on exp() and apply gradient clipping. For reasons please refer to #24.

[Feynman1999]

Thank you for your impressive work. But when I try to recurrent this network(I rewrite the code myself), sometimes the loss will suddenly increase by 10 times. The structure of the network is correct because I can load the pretrained network, so I think there may be some details I didn't notice. Could you tell me what methods you have taken in training to ensure stability?

I have also encountered this issue recently, despite using gradient clipping strategy. How did you solve it?