[Only in Jupyter] OS Error: torch_sparse/_version_cpu.so: undefined symbol XXX (only installed GPU version)

[jasperhyp]

Hi, I am recently building a workplace environment on my Windows 10 laptop. Since I have everything done in my WSL2, I followed the procedure of installing Pytorch and PyG on Linux, i.e., running pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html for PyTorch and pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.10.0+cu113.html for PyG. While PyTorch works smoothly, PyG is suffering from a weird OSError when I use Jupyter Notebook to run it. (But there's no problem when running in command line!)

Error message:

OSError                                   Traceback (most recent call last)
/tmp/ipykernel_578/1803424725.py in <module>
      7 import torch.optim as optim
      8 
----> 9 from torch_geometric.data import Data, GraphSAINTRandomWalkSampler
     10 from torch_geometric.datasets import Planetoid, Entities
     11 from torch_geometric.nn import GCNConv, RGCNConv, GATConv, SAGEConv, JumpingKnowledge, GINConv, DeepGraphInfomax

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_geometric/__init__.py in <module>
      5 from .debug import is_debug_enabled, debug, set_debug
      6 
----> 7 import torch_geometric.data
      8 import torch_geometric.loader
      9 import torch_geometric.transforms

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_geometric/data/__init__.py in <module>
----> 1 from .data import Data
      2 from .hetero_data import HeteroData
      3 from .temporal import TemporalData
      4 from .batch import Batch
      5 from .dataset import Dataset

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_geometric/data/data.py in <module>
      1 from typing import (Optional, Dict, Any, Union, List, Iterable, Tuple,
      2                     NamedTuple, Callable)
----> 3 from torch_geometric.typing import OptTensor, NodeType, EdgeType
      4 from torch_geometric.deprecation import deprecated
      5 

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_geometric/typing.py in <module>
      2 
      3 from torch import Tensor
----> 4 from torch_sparse import SparseTensor
      5 
      6 # Types for accessing data ####################################################

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_sparse/__init__.py in <module>
     13         '_relabel'
     14 ]:
---> 15     torch.ops.load_library(importlib.machinery.PathFinder().find_spec(
     16         f'{library}_{suffix}', [osp.dirname(__file__)]).origin)
     17 

~/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch/_ops.py in load_library(self, path)
    108             # static (global) initialization code in order to register custom
    109             # operators with the JIT.
--> 110             ctypes.CDLL(path)
    111         self.loaded_libraries.add(path)
    112 

~/miniconda3/envs/py39torch110/lib/python3.9/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    380 
    381         if handle is None:
--> 382             self._handle = _dlopen(self._name, mode)
    383         else:
    384             self._handle = handle

OSError: /home/jasperhyp/miniconda3/envs/py39torch110/lib/python3.9/site-packages/torch_sparse/_convert_cpu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv

Specs of my WSL2 environment:

(py39torch110) jasperhyp@LAPTOP-SJR15LFA:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_21:27:02_PDT_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0
(py39torch110) jasperhyp@LAPTOP-SJR15LFA:~$ uname -r
5.10.60.1-microsoft-standard-WSL2
(py39torch110) jasperhyp@LAPTOP-SJR15LFA:~$ cat /usr/local/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 3
#define CUDNN_PATCHLEVEL 1
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#endif /* CUDNN_VERSION_H */

Code to reproduce:

from torch_geometric.data import Data

Conda list:

_libgcc_mutex             0.1                        main
_openmp_mutex             4.5                       1_gnu
argon2-cffi               20.1.0           py39h27cfd23_1
async_generator           1.10               pyhd3eb1b0_0
attrs                     21.2.0             pyhd3eb1b0_0
backcall                  0.2.0              pyhd3eb1b0_0
blas                      1.0                         mkl
bleach                    4.0.0              pyhd3eb1b0_0
ca-certificates           2021.10.26           h06a4308_2
certifi                   2021.10.8        py39h06a4308_0
cffi                      1.14.6           py39h400218f_0
charset-normalizer        2.0.9                    pypi_0    pypi
dbus                      1.13.18              hb2f20db_0
debugpy                   1.5.1            py39h295c915_0
decorator                 5.1.0              pyhd3eb1b0_0
defusedxml                0.7.1              pyhd3eb1b0_0
dgl-cuda11.3              0.7.2                    py39_0    dglteam
entrypoints               0.3              py39h06a4308_0
expat                     2.4.1                h2531618_2
fontconfig                2.13.1               h6c09931_0
freetype                  2.11.0               h70c0345_0
glib                      2.69.1               h5202010_0
googledrivedownloader     0.4                      pypi_0    pypi
gst-plugins-base          1.14.0               h8213a91_2
gstreamer                 1.14.0               h28cd5cc_2
icu                       58.2                 he6710b0_3
idna                      3.3                      pypi_0    pypi
importlib-metadata        4.8.2            py39h06a4308_0
importlib_metadata        4.8.2                hd3eb1b0_0
intel-openmp              2021.4.0          h06a4308_3561
ipykernel                 6.4.1            py39h06a4308_1
ipython                   7.29.0           py39hb070fc8_0
ipython_genutils          0.2.0              pyhd3eb1b0_1
ipywidgets                7.6.5              pyhd3eb1b0_1
isodate                   0.6.1                    pypi_0    pypi
jedi                      0.18.0           py39h06a4308_1
jinja2                    3.0.2              pyhd3eb1b0_0
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9d                   h7f8727e_0
jsonschema                3.2.0              pyhd3eb1b0_2
jupyter                   1.0.0            py39h06a4308_7
jupyter_client            7.0.6              pyhd3eb1b0_0
jupyter_console           6.4.0              pyhd3eb1b0_0
jupyter_core              4.9.1            py39h06a4308_0
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1
ld_impl_linux-64          2.35.1               h7274673_9
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.3.0               h5101ec6_17
libgfortran-ng            7.5.0               ha8ba4b0_17
libgfortran4              7.5.0               ha8ba4b0_17
libgomp                   9.3.0               h5101ec6_17
libpng                    1.6.37               hbc83047_0
libsodium                 1.0.18               h7b6447c_0
libstdcxx-ng              9.3.0               hd4cf53a_17
libuuid                   1.0.3                h7f8727e_2
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.12               h03d6c58_0
markupsafe                2.0.1            py39h27cfd23_0
matplotlib-inline         0.1.2              pyhd3eb1b0_2
mistune                   0.8.4           py39h27cfd23_1000
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py39h7f8727e_0
mkl_fft                   1.3.1            py39hd3c417c_0
mkl_random                1.2.2            py39h51133e4_0
nbclient                  0.5.3              pyhd3eb1b0_0
nbconvert                 6.1.0            py39h06a4308_0
nbformat                  5.1.3              pyhd3eb1b0_0
ncurses                   6.3                  h7f8727e_2
nest-asyncio              1.5.1              pyhd3eb1b0_0
networkx                  2.6.3              pyhd3eb1b0_0
notebook                  6.4.6            py39h06a4308_0
numpy                     1.21.4                   pypi_0    pypi
numpy-base                1.21.2           py39h79a1101_0
openssl                   1.1.1l               h7f8727e_0
packaging                 21.3               pyhd3eb1b0_0
pandas                    1.3.5                    pypi_0    pypi
pandocfilters             1.4.3            py39h06a4308_1
parso                     0.8.2              pyhd3eb1b0_0
pcre                      8.45                 h295c915_0
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    8.4.0                    pypi_0    pypi
pip                       21.2.4           py39h06a4308_0
prometheus_client         0.12.0             pyhd3eb1b0_0
prompt-toolkit            3.0.20             pyhd3eb1b0_0
prompt_toolkit            3.0.20               hd3eb1b0_0
ptyprocess                0.7.0              pyhd3eb1b0_2
pycparser                 2.21               pyhd3eb1b0_0
pygments                  2.10.0             pyhd3eb1b0_0
pyparsing                 3.0.4              pyhd3eb1b0_0
pyqt                      5.9.2            py39h2531618_6
pyrsistent                0.18.0           py39heee7806_0
python                    3.9.7                h12debd9_1
python-dateutil           2.8.2              pyhd3eb1b0_0
pytz                      2021.3                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     22.3.0           py39h295c915_2
qt                        5.9.7                h5867ecd_1
qtconsole                 5.1.1              pyhd3eb1b0_0
qtpy                      1.10.0             pyhd3eb1b0_0
rdflib                    6.0.2                    pypi_0    pypi
readline                  8.1                  h27cfd23_0
requests                  2.26.0                   pypi_0    pypi
scikit-learn              1.0.1                    pypi_0    pypi
scipy                     1.7.1            py39h292c36d_2
send2trash                1.8.0              pyhd3eb1b0_1
setuptools                58.0.4           py39h06a4308_0
sip                       4.19.13          py39h2531618_0
six                       1.16.0             pyhd3eb1b0_0
sqlite                    3.36.0               hc218d9a_0
terminado                 0.9.4            py39h06a4308_0
testpath                  0.5.0              pyhd3eb1b0_0
threadpoolctl             3.0.0                    pypi_0    pypi
tk                        8.6.11               h1ccaba5_0
torch                     1.10.1+cu113             pypi_0    pypi
torch-cluster             1.5.9                    pypi_0    pypi
torch-geometric           2.0.2                    pypi_0    pypi
torch-scatter             2.0.9                    pypi_0    pypi
torch-sparse              0.6.12                   pypi_0    pypi
torch-spline-conv         1.2.1                    pypi_0    pypi
torchaudio                0.10.1+cu113             pypi_0    pypi
torchvision               0.11.2+cu113             pypi_0    pypi
tornado                   6.1              py39h27cfd23_0
tqdm                      4.62.3                   pypi_0    pypi
traitlets                 5.1.1              pyhd3eb1b0_0
typing-extensions         4.0.1                    pypi_0    pypi
tzdata                    2021e                hda174b7_0
urllib3                   1.26.7                   pypi_0    pypi
wcwidth                   0.2.5              pyhd3eb1b0_0
webencodings              0.5.1            py39h06a4308_1
wheel                     0.37.0             pyhd3eb1b0_1
widgetsnbextension        3.5.1            py39h06a4308_0
xz                        5.2.5                h7b6447c_0
yacs                      0.1.8                    pypi_0    pypi
zeromq                    4.3.4                h2531618_0
zipp                      3.6.0              pyhd3eb1b0_0
zlib                      1.2.11               h7b6447c_3

I tried to uninstall and reinstall PyG and PyTorch, with conda, conda-forge, pip, but nothing worked. My guess is that maybe I should downgrade CUDA from 11.4 to 11.3, but Torch for 11.3 worked well under my 11.4 env. Am a bit lost... Would appreciate your help.

[rusty1s]

Can you check if this is indeed a CUDA problem, e.g., by installing the CPU wheels (which should run fine when only operating on CPU tensors)?

[jasperhyp]

Sorry for the late reply, I switched to a server afterwards. But yes, I tried again today and it seems to be fine with CPU.