Training cross-coder error

[zhihui-shao]

Traceback (most recent call last): File "schema_item_classifier.py", line 463, in <module> _train(opt) File "schema_item_classifier.py", line 271, in _train model_outputs = model( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/szh/RESDSQL/utils/classifier_model.py", line 185, in forward = self.table_column_cls( File "/home/shaozhihui/szh/RESDSQL/utils/classifier_model.py", line 106, in table_column_cls encoder_output = self.plm_encoder( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 852, in forward encoder_outputs = self.encoder( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 527, in forward layer_outputs = layer_module( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 411, in forward self_attention_outputs = self.attention( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 338, in forward self_outputs = self.self( File "/home/shaozhihui/anaconda3/envs/RESDSQL/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/shaozhihui/.local/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 234, in forward attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)``

What's the cause, please?

[lihaoyang-ruc]

This could be due to issues within your Python environment.