RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1523244252089/work/torch/lib/THC/generic/THCStorage.cu:58

[sms95]

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1523244252089/work/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "train.py", line 257, in
train()
File "train.py", line 109, in train
dsfd_net = build_net('train', cfg.NUM_CLASSES, args.model)
File "/media/ices18/Data/sms/bishe/tiny_detection/DSFD.pytorch/models/factory.py", line 23, in build_net
return build_net_vgg(phase, num_classes)
File "/media/ices18/Data/sms/bishe/tiny_detection/DSFD.pytorch/models/DSFD_vgg.py", line 350, in build_net_vgg
base = vgg(vgg_cfg, 3)
File "/media/ices18/Data/sms/bishe/tiny_detection/DSFD.pytorch/models/DSFD_vgg.py", line 301, in vgg
conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
File "/home/ices18/software/miniconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 278, in init
False, _pair(0), groups, bias)
File "/home/ices18/software/miniconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 33, in init
out_channels, in_channels // groups, *kernel_size))
File "/home/ices18/software/miniconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/cuda/init.py", line 387, in _lazy_new
return super(_CudaBase, cls).new(cls, *args, **kwargs)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1523244252089/work/torch/lib/THC/generic/THCStorage.cu:58

This issue occur when I train on WIDERFACE

[mafamily2496]

use smaller batch_size