lm_eval not working in the model.py

[HamidShojanazeri]

🐛 Describe the bug

Following the installation README, and running dist_run.py, it fails to import utils from evaluate module. It also seems not being used atm, may as well remove it?

(torchchat)  hamidnazeri@submit-0:~/torchchat$ python dist_run.py --model_name meta-llama/Meta-Llama-3-8B-Instruct
Traceback (most recent call last):
  File "/opt/hpcaas/.mounts/fs-0301404b74c8d22fd/home/hamidnazeri/torchchat/dist_run.py", line 37, in <module>
    from torchchat.cli.builder import _initialize_tokenizer, TokenizerArgs
  File "/opt/hpcaas/.mounts/fs-0301404b74c8d22fd/home/hamidnazeri/torchchat/torchchat/cli/builder.py", line 29, in <module>
    from torchchat.model import Model, ModelArgs, ModelType
  File "/opt/hpcaas/.mounts/fs-0301404b74c8d22fd/home/hamidnazeri/torchchat/torchchat/model.py", line 35, in <module>
    import lm_eval  # noqa 
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/site-packages/lm_eval/__init__.py", line 1, in <module>
    from .evaluator import evaluate, simple_evaluate
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/site-packages/lm_eval/evaluator.py", line 11, in <module>
    import lm_eval.api.metrics
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/site-packages/lm_eval/api/metrics.py", line 168, in <module>
    exact_match = hf_evaluate.load("exact_match")
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/site-packages/evaluate/loading.py", line 751, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/site-packages/evaluate/loading.py", line 76, in import_main_class
    module = importlib.import_module(module_path)
  File "/data/home/hamidnazeri/miniconda3-2/envs/torchchat/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/data/home/hamidnazeri/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/9d3b67e0c429cd7460b2b05aab53419b48eea369b73e1d9f185a56ca90c373d4/exact_match.py", line 86, in <module>
    @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
AttributeError: module 'evaluate' has no attribute 'utils'

Versions

Collecting environment information...
PyTorch version: 2.5.0.dev20240901+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.30.3
Libc version: glibc-2.31

Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1068-aws-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        46 bits physical, 48 bits virtual
CPU(s):                               16
On-line CPU(s) list:                  0-15
Thread(s) per core:                   2
Core(s) per socket:                   8
Socket(s):                            1
NUMA node(s):                         1
Vendor ID:                            GenuineIntel
CPU family:                           6
Model:                                143
Model name:                           Intel(R) Xeon(R) Platinum 8488C
Stepping:                             8
CPU MHz:                              2400.000
BogoMIPS:                             4800.00
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            384 KiB
L1i cache:                            256 KiB
L2 cache:                             16 MiB
L3 cache:                             105 MiB
NUMA node0 CPU(s):                    0-15
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pytorch-triton==3.0.0+dedb7bdf33
[pip3] torch==2.5.0.dev20240901+cu121
[pip3] torchao==0.5.0
[pip3] torchtune==0.0.0
[pip3] torchvision==0.20.0.dev20240901+cu121
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] pytorch-triton            3.0.0+dedb7bdf33          pypi_0    pypi
[conda] torch                     2.5.0.dev20240901+cu121          pypi_0    pypi
[conda] torchao                   0.5.0                    pypi_0    pypi
[conda] torchtune                 0.0.0                    pypi_0    pypi
[conda] torchvision               0.20.0.dev20240901+cu121          pypi_0    pypi

[Jack-Khuu]

Looks like something got jumbled around with the recent pin bumps and refactor. Thanks for flagging we'll take a look

cc @kwen2501

[kwen2501]

I can repro the same error.

[kwen2501]

It seems to be related to this TODO:

torchchat/torchchat/model.py

Lines 33 to 35 in 1980a69

	# TODO: remove this after we figure out where in torchtune an `evaluate` module
	# is being imported, which is being confused with huggingface's `evaluate``.
	import lm_eval # noqa

@larryliu0820 Do you mind taking a look?

I also notice that lm_eval is not used in model.py.

[kwen2501]

Also @lessw2020, wondering why only Distributed got impacted.
Does it have to do with us using HF's Tokenizer?

[lessw2020]

@kwen2501 - we don't use HF Tokenizer anymore. I switched us to TorchChat's tokenizer about a month ago. Our only HF reliance is on SafeTensor weights.

[Jack-Khuu]

The lm_eval is a work around for a bad import state that torchchat ends up in as a result of torchtune deps.

#1228 adds a try catch to unblock dist_run while we continue investigation

[Jack-Khuu]

Thanks again for catching this, and feel free to reopen if a similar error pops up