RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[zwl8979]

When I reproduced softpatch, the noise discriminator reported an error when selecting nearest. The first five categories (bottle, cable, capsule, carpet, grid) ran normally, but when it came to the mental_nut category, an error was reported when printing the Subsampling progress (batchsize=16, no overlap, without nosie augmentation). The same error occurred when the batchsize was set to 4. My graphics card is 4090 (24g memory):

INFO:main:Evaluating dataset [mvtec_hazelnut] (6/15)...
/home/zwl/anaconda3/envs/softpatch/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/zwl/anaconda3/envs/softpatch/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=Wide_ResNet50_2_Weights.IMAGENET1K_V1. You can also use weights=Wide_ResNet50_2_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
INFO:main:Training models (1/1)
Computing support features...: 100%|██████████| 27/27 [00:15<00:00, 1.80it/s]
Traceback (most recent call last):
File "/home/zwl/Zwl/SoftPatch-main/main.py", line 555, in
run(args)
File "/home/zwl/Zwl/SoftPatch-main/main.py", line 369, in run
coreset.fit(dataloaders["training"])
File "/home/zwl/Zwl/SoftPatch-main/src/softpatch.py", line 225, in fit
self._fill_memory_bank(training_data)
File "/home/zwl/Zwl/SoftPatch-main/src/softpatch.py", line 272, in _fill_memory_bank
sample_features, sample_indices = self.featuresampler.run(features)
File "/home/zwl/Zwl/SoftPatch-main/src/sampler.py", line 111, in run
reduced_features = self._reduce_features(features)
File "/home/zwl/Zwl/SoftPatch-main/src/sampler.py", line 90, in _reduce_features
return mapper(features)
File "/home/zwl/anaconda3/envs/softpatch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zwl/anaconda3/envs/softpatch/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

This is the parameter configuration of my main function（No overlap - without augumentation）
--gpu
0
--seed
0
--results_path
result
--log_project
MVTecAD-wideresnet50-noise0.1
--log_group
No-overlap-SW-NA-nearest_0
--save_segmentation_images
--sampler_name
approx_greedy_coreset
--sampling_ratio
0.1
--faiss_on_gpu
--faiss_num_workers
8
--weight_method
nearest
--threshold
0.15
--lof_k
6
--dataset
mvtec
--data_path
./MVTecAD
--subdatasets
bottle
cable
capsule
carpet
grid
hazelnut
leather
metal_nut
pill
screw
tile
toothbrush
transistor
wood
zipper
--batch_size
16
--resize
256
--imagesize
224
--noise
0.1
--fold
0

Have you ever encountered this problem? How did you solve it? @jam-cc

[jam-cc]

I haven't encountered this issue before, but based on the information you've provided, it seems likely that the error might be related to insufficient GPU memory, especially since some categories are running without issues.

However, under the default parameter settings, your device should have adequate memory. It's possible that other programs are occupying the GPU memory, or there might be some memory that hasn't been cleared properly.

[zwl8979]

I haven't encountered this issue before, but based on the information you've provided, it seems likely that the error might be related to insufficient GPU memory, especially since some categories are running without issues.

However, under the default parameter settings, your device should have adequate memory. It's possible that other programs are occupying the GPU memory, or there might be some memory that hasn't been cleared properly.

Thanks for your help. I tried to reduce the batch size to 4, but still got the same error. When I tried to set --noise to 0, it worked fine. The screenshot shows the status of my GPU running. The GPU occupancy is about 6G, which is sufficient; but the GPU utilization rate reached 98% when calculating the nearest neighbor of the feature before core set sampling. In addition, I tried a lower abnormal sample noise ratio and found that this problem occurred when --noise reached 0.025. Is there any good way to solve this problem?
@jam-cc

[zwl8979]

I haven't encountered this issue before, but based on the information you've provided, it seems likely that the error might be related to insufficient GPU memory, especially since some categories are running without issues.
However, under the default parameter settings, your device should have adequate memory. It's possible that other programs are occupying the GPU memory, or there might be some memory that hasn't been cleared properly.

Thanks for your help. I tried to reduce the batch size to 4, but still got the same error. When I tried to set --noise to 0, it worked fine. The screenshot shows the status of my GPU running. The GPU occupancy is about 6G, which is sufficient; but the GPU utilization rate reached 98% when calculating the nearest neighbor of the feature before core set sampling. In addition, I tried a lower abnormal sample noise ratio and found that this problem occurred when --noise reached 0.025. Is there any good way to solve this problem? @jam-cc

Try the following yourself:
try:
reduced_features = self._reduce_features(features)
torch.cuda.synchronize() # Force waiting for GPU completion
print("Reduce features done. shape:", reduced_features.shape)
except RuntimeError as e:
print("Error in reduce_features:", e)
raise
The problem is most likely to occur in the call of reduced_features = self._reduce_features(features):
reduced_features = self._reduce_features(features) call when trying the following:
def _reduce_features(self, features)**:
if features.shape[1] == self.dimension_to_project_features_to:
return features
mapper = torch.nn.Linear(
features.shape[1], self.dimension_to_project_features_to, bias=False
_ = mapper.to(self.device)
features = features.to(self.device)
return mapper(features)
Perform the linear mapping stage with the mapper(features) operation, do you have any good suggestions and solutions? Thank you! @jam-cc