[ASSERT ]：failed: CUDA Runtime Error: invalid device ordinal

[tianqixun]

Hello, I have just completed the installation. An assert error occurred when running the first example with :python dreamplace/Placer.py test/ispd2005/adaptec1.json
[DEBUG ] Standard cell legalization blank_num_bins_y = 890
[DEBUG ] Standard cell legalization iteration 0 with 1x1 bins
[DEBUG ] Standard cell legalization #bin_blanks
[DEBUG ] #bin_objs = 10792
[INFO ] Standard cell legalization legalizeBin takes 352.296 ms
[DEBUG ] Standard cell legalization num_unplaced_cells = 0
[INFO ] Greedy legalization takes 328.36 ms
[INFO ] Legality check takes 16.6944 ms
[DEBUG ] average displace = 14.2921
[INFO ] Abacus legalization takes 28.3865 ms
[INFO ] Legality check takes 15.8973 ms
[INFO ] DREAMPlace - legalization takes 0.405 seconds
[INFO ] DREAMPlace - additional sta after legalization
[INFO ] DREAMPlace - iteration 612, wHPWL 7.395942E+07, time 7.434ms
[INFO ] DREAMPlace - Start ABCDPlace for refinement
[INFO ] Legality check takes 16.0258 ms
[INFO ] DREAMPlace - ABCDPlace input legal flag = 1
[DEBUG ] 4-reorder
[DEBUG ] 24 permutations
[DEBUG ] initializing CPU DB takes 16.1105 ms
[DEBUG ] computing independent rows takes 52.9867 ms
[DEBUG ] computing reorder instances takes 0.843592 ms
[DEBUG ] initializing CUDA state takes 1.80678 ms
[INFO ] initial hpwl = 74043808.000
[DEBUG ] enumeration takes 2341.41 ms for 216 runs, average 10.8399 ms
[DEBUG ] apply_reorder takes 599.096 ms for 216 runs, average 2.77359 ms
[INFO ] Iteration time(ms) 2940.55
[INFO ] iteration 0: hpwl 74043808.000 => 73820176.000 (imp. 0.302027%)
[DEBUG ] enumeration takes 2127.48 ms for 216 runs, average 9.84944 ms
[DEBUG ] apply_reorder takes 544.53 ms for 216 runs, average 2.52097 ms
[INFO ] Iteration time(ms) 2672.06
[INFO ] iteration 1: hpwl 74043808.000 => 73778664.000 (imp. 0.358091%)
[DEBUG ] destroying CUDA state takes 681.992 ms
[DEBUG ] destroying CPU state takes 0.003164 ms
[INFO ] Kernel time: 5621.43 ms
[INFO ] K-reorder time: 6375.3 ms
[INFO ] Legality check takes 21.1534 ms
[INFO ] DREAMPlace - K-Reorder legal flag = 1
[INFO ] diamond search grid size 64, sequence length 2048
[INFO ] initializing GPU memory takes 13.4123 ms
[INFO ] initial hpwl 0
[ASSERT ] /u3disk/tianqx24/Desktop/cudacuda/DREAMPlace/dreamplace/ops/independent_set_matching/../independent_set_matching/src/select.cuh:39: void DreamPlace::select(const int*, int*, int, int*, int*): Assertion `cub::DeviceScan::ExclusiveSum(d_temp_storage, temp_storage_bytes, d_flags, prefix_sum, length) == cudaSuccess' failed: CUDA Runtime Error: invalid device ordinal

I set up a virtual environment using conda and installed CUDA version 11.8 and GPU version 7.5 in the environment
When I run nvidia-smi, it shows:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A800 80GB PCIe Off | 00000000:31:00.0 Off | 0 |
| N/A 43C P0 88W / 300W | 20241MiB / 81920MiB | 92% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A800 80GB PCIe Off | 00000000:4B:00.0 Off | 0 |
| N/A 38C P0 74W / 300W | 22722MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A800 80GB PCIe Off | 00000000:CA:00.0 Off | 0 |
| N/A 55C P0 125W / 300W | 26214MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+

I have tried the following methods:
①CUDA_VISIBLE_DEVICES=0 python dreamplace/Placer.py test/ispd2005/adaptec1.json
However, none of the above methods work very well

By the way, such an error occurs whether I install the master version or the 4.1.0 version

[limbo018]

We recently found CUDA 11.8 may have backward compatibility issues. Could you try CUDA 11.5 or 11.6? I don't see any problem on adaptec1 on my side.

[tianqixun]

I have tried these two versions and also replaced the corresponding pytorch to the corresponding version, but the same error still occurs. By the way, when I was compiling with make using CUDA11.8, it prompted that g++ couldn't be found. So I also changed g++ to version 7.5. However, I think it's not because of this either.

[limbo018]

I cannot reproduce the problem when using PyTorch 1.7 and CUDA 11.5 with RTX 2070. My GPU driver version is 530.30.02.

If the error only happens on the independent set matching op, you can directly disable it in the BasicPlace.py.