Something about distributed sampling and official example's environment.

[skdbsxir]

Hello, and also very thanks for excellent works :)

I fixed some lines of distributed_sampling.py and tested in my workstation environment. (My fixed code is here)

My workstation environment is like below.

OS : Ubuntu 20.04 LTS
Kernel : 1.15.0-58-generic
CPU : AMD Ryzen Threadripper PRO 5955WX 16-Cores
RAM : 128GB
GPU : NVIDIA  RTX A5000 * 4 (24GB RAM each)
CUDA : 11.7

And my main packages are like below.

torch==1.13.1
pyg-lib==0.1.0+pt113cu117
torch-scatter==2.1.0+pt113cu117
torch-sparse==0.6.16+pt113cu117
torch-cluster==1.6.0+pt113cu117
torch-spline-conv==1.2.1+pt113cu117
torch-geometric==2.2.0

As similar with PyG's official example, I used Reddit dataset, and used SAGEConv, NeighborLoader, DistributedDataParallel.

But in each experiments, I used 3-layer SAGEConv with num_neighbors as [15,10,5], [25,15,5], [35,20,5] and hidden_channels=128, and batch_size=1024.

When I was running num_neighbors=[35,20,5], one of my GPU has broken, and nvidia-smi command not worked.

After rebooted my workstation, device was seemed to be recovered, since nvidia-smi command worked normally.

But when I ran experiment with num_neighbors=[15,10,5], GPU has broken again (but before start training, not during training) and same situation (like above) was appeared.

I ran nvidia-bug-report.sh and checked the error, the error was like below.

NVRM: Xid (PCI:0000:42:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
NVRM: GPU 0000:42:00.0: GPU has fallen off the bus.

After some web searching, I found it was about some HW issues, from here and here.

I sent my workstation to company which assembled it, and they said it looks like my code didn't thought about GPU specs.

I think it was my workstation's main board issue (especially lack of power), but company said there seems to be no HW issues when they tested with gpu-burn.

So, my main question is :

1. Does my experiment setup was seems to be over-specced to GPUs?
   • But before, I ran num_neighbors=[30,20,10] with hidden_channels=256 and batch_size=1024, experiment has exited normally.
2. In which environment was the original example (distributed_sampling.py) run? I want to compare if it was a problem in my experiment setup, which doesn't consider my GPU specs.

Thanks a lot in advance for any! :)

[LukeLIN-web]

It seems that there are some problems with your Hardware.
I installed current master branch and it works well.

torch-scatter             2.1.0
torch-sparse              0.6.16
pyg-lib                   0.1.0+pt111cu113
torch                     1.11.0
torch-geometric           2.3.0            /root/share/pytorch_geometric

[skdbsxir]

Thanks for reply!

Can I check your hardware specs, especially GPUs?

And did you ran the official example, not fixing arguments such as num_neighbors, hidden_channels, num_layers?

I think my overall hardware is the main problem (maybe except GPUs), but company says hardware seems no problem.
(They said they tested GPUs with gpu-burn program, but there doesn't occurs any problems.)

Does my fixed code seems to be causing some issues?

[skdbsxir]

Cross-checked with company, using another GPUs (A4000 * 4), found my code ran normally.

And when using my 2 GPUs(A500 * 2), code ran normally, but failed when using my 3 GPUs.

Also I checked many other cases from NVIDIA Developer forum and talked with company, they concluded it seems main board's power issue 😵

Sorry for discussing my own HW issue, at first I thought my modified code and arguments(num_neighbors, batch_size) was over-set, not considering HW specs.

I'll close this discussion, thanks!