I got an error in open lm installation

[orhanerday]

Environment

• Transformers and Tokenizer versions: transformers-4.43.3 tokenizers-0.19.1
• Platform: Python 3 Google Compute Engine backend (GPU)
• Hardware accelerator: A100 GPU
• Python version: Python 3.10.12

I got an error in open lm installation

I ran
pip install git+https://github.com/mlfoundations/open_lm.git

Here is the error traceback I received:

Collecting pyyaml>=5.1 (from datasets->open_lm==0.0.34)
Using cached PyYAML-5.4.1.tar.gz (175 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error

[orhanerday]

Here is the whole execution process

Collecting git+https://github.com/mlfoundations/open_lm.git
  Cloning https://github.com/mlfoundations/open_lm.git to /tmp/pip-req-build-gdj_yt4y
  Resolved https://github.com/mlfoundations/open_lm.git to commit 9bb92ef1689333534b7057942a20d18a46d1fa52
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (2.3.1+cu121)
Collecting xformers>=0.0.22 (from open_lm==0.0.34)
  Using cached xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting tiktoken (from open_lm==0.0.34)
  Using cached tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting wandb (from open_lm==0.0.34)
  Using cached wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Collecting webdataset (from open_lm==0.0.34)
  Using cached webdataset-0.2.86-py3-none-any.whl.metadata (29 kB)
Collecting pandas==2.1.4 (from open_lm==0.0.34)
  Using cached pandas-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (2024.6.1)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (4.66.4)
Collecting jsonlines (from open_lm==0.0.34)
  Using cached jsonlines-4.0.0-py3-none-any.whl.metadata (1.6 kB)
Collecting boto3 (from open_lm==0.0.34)
  Using cached boto3-1.34.149-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (9.4.0)
Collecting zstandard (from open_lm==0.0.34)
  Using cached zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting pysimdjson (from open_lm==0.0.34)
  Using cached pysimdjson-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Requirement already satisfied: cloudpathlib in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (0.18.1)
Collecting datasets (from open_lm==0.0.34)
  Using cached datasets-2.20.0-py3-none-any.whl.metadata (19 kB)
Collecting multiprocess>=0.70.11 (from open_lm==0.0.34)
  Using cached multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
Collecting dill (from open_lm==0.0.34)
  Using cached dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (0.23.5)
Collecting pre-commit (from open_lm==0.0.34)
  Using cached pre_commit-3.7.1-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting ray[all] (from open_lm==0.0.34)
  Using cached ray-2.33.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting loguru (from open_lm==0.0.34)
  Using cached loguru-0.7.2-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (4.42.4)
Collecting s3fs (from open_lm==0.0.34)
  Using cached s3fs-2024.6.1-py3-none-any.whl.metadata (1.6 kB)
Collecting wikipedia (from open_lm==0.0.34)
  Using cached wikipedia-1.4.0.tar.gz (27 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: ipython in /usr/local/lib/python3.10/dist-packages (from open_lm==0.0.34) (7.34.0)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.23.5-py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: numpy<2,>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (1.25.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas==2.1.4->open_lm==0.0.34) (2024.1)
Collecting torch (from open_lm==0.0.34)
  Using cached torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (1.13.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->open_lm==0.0.34) (3.1.4)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch->open_lm==0.0.34)
  Using cached nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->open_lm==0.0.34)
  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->open_lm==0.0.34)
  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch->open_lm==0.0.34)
  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->open_lm==0.0.34)
  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->open_lm==0.0.34)
  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch->open_lm==0.0.34)
  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch->open_lm==0.0.34)
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton==3.0.0 (from torch->open_lm==0.0.34)
  Using cached triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch->open_lm==0.0.34)
  Using cached nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting botocore<1.35.0,>=1.34.149 (from boto3->open_lm==0.0.34)
  Using cached botocore-1.34.149-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->open_lm==0.0.34)
  Using cached jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->open_lm==0.0.34)
  Using cached s3transfer-0.10.2-py3-none-any.whl.metadata (1.7 kB)
Collecting pyarrow>=15.0.0 (from datasets->open_lm==0.0.34)
  Using cached pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (0.6)
Collecting requests>=2.32.2 (from datasets->open_lm==0.0.34)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting xxhash (from datasets->open_lm==0.0.34)
  Using cached xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting fsspec (from open_lm==0.0.34)
  Using cached fsspec-2024.5.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (3.9.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets->open_lm==0.0.34) (6.0.1)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (71.0.4)
Collecting jedi>=0.16 (from ipython->open_lm==0.0.34)
  Using cached jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (4.4.2)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (5.7.1)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (3.0.47)
Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (2.16.1)
Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.2.0)
Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (0.1.7)
Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython->open_lm==0.0.34) (4.9.0)
Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->open_lm==0.0.34) (23.2.0)
Collecting torchmetrics<1.3.3,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.2-py3-none-any.whl.metadata (19 kB)
Collecting torch-optimizer<0.4,>=0.3.0 (from mosaicml->open_lm==0.0.34)
  Using cached torch_optimizer-0.3.0-py3-none-any.whl.metadata (55 kB)
Requirement already satisfied: torchvision<0.18.2,>=0.13.1 in /usr/local/lib/python3.10/dist-packages (from mosaicml->open_lm==0.0.34) (0.18.1+cu121)
INFO: pip is looking at multiple versions of mosaicml to determine which version is compatible with other requirements. This could take a while.
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.23.4-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.23.0-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.18.1,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.18.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.22.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.21.3-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.17.3,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.17.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
INFO: pip is still looking at multiple versions of mosaicml to determine which version is compatible with other requirements. This could take a while.
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.21.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.21.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<1.3.2,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.1-py3-none-any.whl.metadata (19 kB)
Collecting torchvision<0.17.2,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.17.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.21.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.20.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.20.0-py3-none-any.whl.metadata (28 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached mosaicml-0.19.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<1.3.1,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.3.0.post0-py3-none-any.whl.metadata (20 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.19.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.18.2-py3-none-any.whl.metadata (27 kB)
Collecting torchvision<0.17,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.18.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.17.2-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<1.1,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-1.0.3-py3-none-any.whl.metadata (21 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.17.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.17.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.16.4-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.16.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.16.0-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.16,>=0.13.1 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.15.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<0.12,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.11.4-py3-none-any.whl.metadata (15 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.15.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.14.1-py3-none-any.whl.metadata (28 kB)
Collecting torchmetrics<0.11.4,>=0.10.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.11.3-py3-none-any.whl.metadata (15 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.14.0-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.5-py3-none-any.whl.metadata (28 kB)
Collecting torchvision<0.15,>=0.11.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchvision-0.14.1-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.13.4-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.3-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.2-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.13.1-py3-none-any.whl.metadata (28 kB)
  Using cached mosaicml-0.12.1-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<0.10.0,>=0.7.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.9.3-py3-none-any.whl.metadata (17 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.12.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.11.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.11.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.10.1-py3-none-any.whl.metadata (27 kB)
Collecting torchmetrics<0.8,>=0.7.0 (from mosaicml->open_lm==0.0.34)
  Using cached torchmetrics-0.7.3-py3-none-any.whl.metadata (20 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.10.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.9.0-py3-none-any.whl.metadata (27 kB)
Collecting torch-optimizer<0.2,>=0.1.0 (from mosaicml->open_lm==0.0.34)
  Using cached torch_optimizer-0.1.0-py3-none-any.whl.metadata (53 kB)
Collecting mosaicml (from open_lm==0.0.34)
  Using cached mosaicml-0.8.2-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.8.1-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.8.0-py3-none-any.whl.metadata (27 kB)
  Using cached mosaicml-0.7.1-py3-none-any.whl.metadata (26 kB)
Collecting pyyaml>=5.1 (from datasets->open_lm==0.0.34)
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error

[DimaDidmanidze]

Getting the same error.