Gradient clipping

[mikecmpbll]

Prevent exploding gradients with gradient clipping .. Sample gradients at beginning of training (perhaps?) and get an average of the expected gradient threshold. If subsequent gradients are above this threshold then scale them down:

if grad.abs > threshold
   grad *= threshold/grad.abs
end