cuda error

[mc171819]

hi, when i run your train.py, it comes out an error:
2021-08-13 15:43:07,291 INFO Start training voxel_rcnn/voxel_rcnn_car(default)
epochs: 0%| | 0/80 [00:00<?, ?it/sError!: 0%| | 0/3741 [00:00<?, ?it/s]
Error!
epochs: 0%| | 0/80 [01:15<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 198, in
main()
File "train.py", line 170, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/data/mc_data/Voxel-R-CNN-main/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/data/mc_data/Voxel-R-CNN-main/tools/train_utils/train_utils.py", line 38, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/home/mc/Project/OpenPCDet/pcdet/models/init.py", line 42, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/mc/Project/OpenPCDet/pcdet/models/detectors/voxel_rcnn.py", line 11, in forward
batch_dict = cur_module(batch_dict)
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/mc/Project/OpenPCDet/pcdet/models/roi_heads/voxelrcnn_head.py", line 227, in forward
targets_dict = self.assign_targets(batch_dict)
File "/home/mc/Project/OpenPCDet/pcdet/models/roi_heads/roi_head_template.py", line 104, in assign_targets
targets_dict = self.proposal_target_layer.forward(batch_dict)
File "/home/mc/Project/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 33, in forward
batch_dict=batch_dict
File "/home/mc/Project/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 101, in sample_rois_for_rcnn
gt_boxes=cur_gt[:, 0:7], gt_labels=cur_gt[:, -1].long()
File "/home/mc/Project/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 223, in get_max_iou_with_same_class
iou3d = iou3d_nms_utils.boxes_iou3d_gpu(cur_roi, cur_gt) # (M, N)
File "/home/mc/Project/OpenPCDet/pcdet/ops/iou3d_nms/iou3d_nms_utils.py", line 71, in boxes_iou3d_gpu
overlaps_h = torch.clamp(min_of_max - max_of_min, min=0)
RuntimeError: CUDA error: invalid device function
Traceback (most recent call last):
File "/opt/anaconda3/envs/objfuse/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/opt/anaconda3/envs/objfuse/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in
main()
File "/opt/anaconda3/envs/objfuse/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/anaconda3/envs/objfuse/bin/python', '-u', 'train.py', '--local_rank=0', '--launcher', 'pytorch', '--cfg_file', 'cfgs/voxel_rcnn/voxel_rcnn_car.yaml', '--epochs', '80', '--workers', '8']' died with <Signals.SIGSEGV: 11>.

i use pytorch1.4,cudatoolkit=10.1,gpu is 2080ti. canyou give me some advice?

[djiajunustc]

Hi @mc171819 ,

This error hasn't occurred to me.
I suggest you run the code with the docker image I provide.

[mc171819]

Hi @mc171819 ,

This error hasn't occurred to me.
I suggest you run the code with the docker image I provide.

hi， i tried using the docker image you provide, but it still doesn't work. i wonder if i mistake something. can you show me the concret step to use the docker image?

[mc171819]

hi, i wonder how the train_dataset is obtained. whatever change i make in build_dataloader in pcdet/dataset/__init__, or kitti/processor/data_processor it doesn't work, even if i make a bug. can you tell me why? At 2021-08-13 19:11:07, "djiajunustc" ***@***.***> wrote: Hi @mc171819 , This error hasn't occurred to me. I suggest you run the code with the docker image I provide. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.