Bounding Box Regression task not learning, it has constant loss

[MohsinTariq10]

I am implementing VPGNET in keras. Task 2,3,4 are classification tasks, so they are learning and inferring okay. But grid box regression task isn't learning. It has a constant loss. I am using linear activation function on the last layer and using Mean Absolute error L1 loss function for bounding box regression task as per paper. I am not normalising any bounding box coordinates, so their range is 0-640 (width) and 0-480 height. The training MAE loss starts off really high, 17.5 to be exact and then stays there.