Accuracy for ntu60 RGB

[dhruvgrover1251]

Hello,

I tried training r3d_18 and r3d_34 on ntu60 RGB videos, I got 3% lower test accuracy on r3d_34 as compared to r3d_18.
I am averaging accuracy over 4 clips of 8 frames each.
Using SGD with an initial LR of 0.001 and dividing by 10 of validation loss saturates.
Using a weight decay of 0.001.
I am using random initialization (not using pre-trained models)

What could be a possible reason for it?

Thank You!