Out of memory, even only batch size = 4

[vince2003]

Hi Chrischoy,
Thank you for your great work. However, when I train on S3DIS with the batch size is only 4, it has out of memory problem. My GPU: titan v100(12G)

You can see the figure. When the training is at epoch 15, it seems the leak of memory. Can you fix that error?
Thank you!

[chrischoy]

Theres no memory leakage. Thank you

[vince2003]

So, what is the error? How can I fix it? Thank you

[chrischoy]

There could be multiple reasons, the pytorch memory manager could have potentially fragmented the GPU too much and couldn't allocate larger space.
Also, Stanford dataset contains extremely large scenes (e.g. auditorium) and random crops might have cropped chunk too large for the memory.