Singularity ImportError: cannot import name 'SemLock' from '_multiprocessing'

[cagatayalici]

Hi!

I've built the singularity container with the definition file provided in the repository.
I uploaded the necessary files into slurm cluster.

Files are arranged as followed:
/projects/
├── container
│ ├── pyg.sif
├── main.py

I run the the job by:
singularity exec --nv --no-home -B /projects/:/home /projects/container/pyg.sif python /home/main.py

However I get the following error:

Traceback (most recent call last):
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/opt/pyenv/versions/3.7.2/lib/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/main.py", line 44, in <module>
    from torch_geometric.loader import DataLoader
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/torch_geometric/__init__.py", line 2, in <module>
    import torch_geometric.nn
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/torch_geometric/nn/__init__.py", line 9, in <module>
    from .models import *  # noqa
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/torch_geometric/nn/models/__init__.py", line 2, in <module>
    from .node2vec import Node2Vec
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/torch_geometric/nn/models/node2vec.py", line 3, in <module>
    from sklearn.linear_model import LogisticRegression
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 25, in <module>
    from . import _joblib
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/utils/_joblib.py", line 7, in <module>
    import joblib
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/__init__.py", line 120, in <module>
    from .parallel import Parallel
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/parallel.py", line 30, in <module>
    from .externals import loky
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/externals/loky/__init__.py", line 11, in <module>
    from .backend.context import cpu_count
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/externals/loky/backend/__init__.py", line 13, in <module>
    from multiprocessing import synchronize
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/multiprocessing/synchronize.py", line 32, in <module>
    " synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
srun: error: srvdgx1: task 0: Exited with exit code 1

I searched on internet but could not find anything.
I appreciate any help.
Thank you!

[rusty1s]

Thanks for sharing this. I'm not that familiar with SLURM but it looks like the error is coming from sklearn's LogisticRegression implementation rather than from PyG directly. Can you confirm?

[cagatayalici]

Hi Matthias!

Thank you very much for your answer.
Yes, it is indeed, you are right.

I executed the singularity image only for from sklearn.linear_model import LogisticRegression.
I got the same error:

Traceback (most recent call last):
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/opt/pyenv/versions/3.7.2/lib/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/code/debug2.py", line 1, in <module>
    from sklearn.linear_model import LogisticRegression
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/__init__.py", line 82, in <module>
    from .base import clone
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/base.py", line 17, in <module>
    from .utils import _IS_32BIT
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 25, in <module>
    from . import _joblib
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/sklearn/utils/_joblib.py", line 7, in <module>
    import joblib
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/__init__.py", line 120, in <module>
    from .parallel import Parallel
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/parallel.py", line 30, in <module>
    from .externals import loky
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/externals/loky/__init__.py", line 11, in <module>
    from .backend.context import cpu_count
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/site-packages/joblib/externals/loky/backend/__init__.py", line 13, in <module>
    from multiprocessing import synchronize
  File "/opt/pyenv/versions/3.7.2/lib/python3.7/multiprocessing/synchronize.py", line 32, in <module>
    " synchronization primitives needed will not" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I will look at their repo.
Thank you!