How to solve the error "cuda runtime error (98) : invalid device function" when run the borderDet?

[fengqian-wei]

I first install cvpods and success to train retinaNet.
However, I face the error in border_align when I train borderDet.
Now I don't know how to fix it.

发生异常: RuntimeError
cuda runtime error (98) : invalid device function at /home//weizhiwei/work/cvpods/cvpods/layers/csrc/border_align/border_align_kernel.cu:202
File "/home/weizhiwei/work/cvpods/cvpods/layers/border_align.py", line 15, in forward
output = _C.border_align_forward(input, boxes, wh, pool_size)
File "/home/weizhiwei/work/cvpods/cvpods/layers/border_align.py", line 42, in forward
output = border_align(feature, boxes, wh, self.pool_size)
File "/home/weizhiwei/work/cvpods/BorderDet/playground/detection/coco/borderdet/borderdet.res50.fpn.coco.800size.1x/borderdet.py", line 809, in forward
ltrb_conv = self.border_align(feature, boxes)
File "/home/weizhiwei/work/cvpods/BorderDet/playground/detection/coco/borderdet/borderdet.res50.fpn.coco.800size.1x/borderdet.py", line 734, in forward
border_cls_conv = self.border_cls_subnet(cls_subnet, align_boxes, wh)
File "/home/weizhiwei/work/cvpods/BorderDet/playground/detection/coco/borderdet/borderdet.res50.fpn.coco.800size.1x/borderdet.py", line 147, in forward
) = self.head(features, shifts)
File "/home/weizhiwei/work/cvpods/cvpods/engine/base_runner.py", line 185, in run_step
loss_dict = self.model(data)
File "/home/weizhiwei/work/cvpods/cvpods/engine/base_runner.py", line 84, in train
self.run_step()
File "/home/weizhiwei/work/cvpods/cvpods/engine/runner.py", line 271, in train
super().train(self.start_iter, self.start_epoch, self.max_iter)
File "/home/weizhiwei/work/cvpods/BorderDet/playground/detection/coco/borderdet/borderdet.res50.fpn.coco.800size.1x/train_net.py", line 96, in main
runner.train()
File "/home/weizhiwei/work/cvpods/cvpods/engine/launch.py", line 56, in launch
main_func(*args)
File "/home/weizhiwei/work/cvpods/BorderDet/playground/detection/coco/borderdet/borderdet.res50.fpn.coco.800size.1x/train_net.py", line 110, in
args=(args,),

[fengqian-wei]

error： when train in single GPU, Default process group is not initialized
solution：https://blog.csdn.net/m0_37568067/article/details/109785209

[Maycbj]

https://xiulian.blog.csdn.net/article/details/111035882
It works well on 2080Ti and V100. Maybe you should follow the methods as follows:
https://blog.csdn.net/m0_38007695/article/details/107065617

[Maycbj]

error： when train in single GPU, Default process group is not initialized
solution：https://blog.csdn.net/m0_37568067/article/details/109785209

yeah, it is a well known bug when training on single GPU. We will fix the Default process group initialized.

[wwwyyk]

have you solved this problem? I can't solve the problem too.