train question #59
Description
When I run train.py in kitti6 dataset, I always encounter the following bugs, and I can't find the reason. Can you help me take a look?
ds_train: 3000
ds_val: 82
LOAD NET CHECKPOINT /home/wenbo/code_cy/READ-main/data/logs_offical/path/UNet_stage_0_epoch_net.pth
EPOCH 0
> TRAIN
EVAL MODE IN TRAIN
model parameters: 30193988
running on datasets [0]
model3d {'rgb': array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
...,
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]], dtype=float32), 'normals': None, 'uv2d': array([[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]], dtype=float32), 'faces': array([0, 1, 2], dtype=uint32), 'xyz': TrackedArray([[ -21.45231247, 13.24265003, -207.19848633],
[ -21.45231247, 13.24265003, -207.19848633],
[ -21.84580994, 14.13696384, -209.91720581],
...,
[ -30.93203926, 15.21014023, 684.93322754],
[ -30.93203926, 15.21014023, 684.93322754],
[ -30.93203926, 15.21014023, 684.93322754]]), 'xyz_c': TrackedArray([[0.41255814, 0.4299459 , 0.02919084],
[0.41255814, 0.4299459 , 0.02919084],
[0.41 , 0.44497895, 0.02623283],
...,
[0.3509302 , 0.46301866, 0.9998443 ],
[0.3509302 , 0.46301866, 0.9998443 ],
[0.3509302 , 0.46301866, 0.9998443 ]], dtype=float32), 'uv1d': array([ 0, 1, 2, ..., 2547319, 2547320, 2547321])}
proj_matrix was not set
total parameters: 50572564[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
Traceback (most recent call last):
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/multiprocessing/queues.py", line 114, in get
raise Empty
_queue.Empty
The above exception was the direct cause of the following exception:Traceback (most recent call last):
File "/home/wenbo/code_cy/READ-main/train.py", line 546, in <module>
train_loss = run_train(epoch, pipeline, args, iter_cb)
File "/home/wenbo/code_cy/READ-main/train.py", line 281, in run_train
return run_epoch(pipeline, 'train', epoch, args, iter_cb=iter_cb)
File "/home/wenbo/code_cy/READ-main/train.py", line 256, in run_epoch
run_sub(dl, extra_optimizer)
File "/home/wenbo/code_cy/READ-main/train.py", line 146, in run_sub
for it, data in enumerate(dl):
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
success, data = self._try_get_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 8638, 8658, 8678) exited unexpectedly
deleting buffers...
Process finished with exit code 1 @JOP-Lee Looking forward to your reply！