[BUG] truthfulqa requires extended_tasks dependencies because registry eagerly imports all extended tasks

[mcavdar]

Describe the bug

I ran into an issue when trying to evaluate a simple leaderboard task (truthfulqa:mc) using a local Hugging Face model.

Even though truthfulqa:mc should only require the base dependencies (transformers, datasets, torch), the run crashes with:

ModuleNotFoundError: No module named 'emoji'

Looking at the traceback:

lighteval/tasks/registry.py imports lighteval.tasks.extended

__init__.py inside extended/ eagerly imports all extended benchmarks (ifbench, ifeval, hle, lcb, etc.)

ifbench → instructions.py → requires emoji

So when the registry is loaded, all extended tasks and their dependencies are imported at startup, regardless of whether the user actually requested them.

Full traceback:

╭──────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────────────╮
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/main_accelerate.py:90 in accelerate                                                                                       │
│                                                                                                                                                                                           │
│    87 │                                                                                                                                                                                   │
│    88 │   from lighteval.logging.evaluation_tracker import EvaluationTracker                                                                                                              │
│    89 │   from lighteval.models.abstract_model import ModelConfig                                                                                                                         │
│ ❱  90 │   from lighteval.models.transformers.adapter_model import AdapterModelConfig                                                                                                      │
│    91 │   from lighteval.models.transformers.delta_model import DeltaModelConfig                                                                                                          │
│    92 │   from lighteval.models.transformers.transformers_model import TransformersModelConfig                                                                                            │
│    93 │   from lighteval.models.transformers.vlm_transformers_model import VLMTransformersMode                                                                                            │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/models/transformers/adapter_model.py:31 in <module>                                                                       │
│                                                                                                                                                                                           │
│    28 import transformers                                                                                                                                                                 │
│    29 from transformers import AutoModelForCausalLM                                                                                                                                       │
│    30                                                                                                                                                                                     │
│ ❱  31 from lighteval.models.transformers.transformers_model import TransformersModel, Transfor                                                                                            │
│    32 from lighteval.models.utils import _get_dtype                                                                                                                                       │
│    33 from lighteval.utils.imports import is_package_available, requires                                                                                                                  │
│    34                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/models/transformers/transformers_model.py:56 in <module>                                                                  │
│                                                                                                                                                                                           │
│     53 from lighteval.models.utils import _get_dtype, _get_model_sha, _simplify_name, uses_chat                                                                                           │
│     54 from lighteval.tasks.prompt_manager import PromptManager                                                                                                                           │
│     55 from lighteval.tasks.requests import Doc, SamplingMethod                                                                                                                           │
│ ❱   56 from lighteval.utils.cache_management import SampleCache, cached                                                                                                                   │
│     57 from lighteval.utils.imports import (                                                                                                                                              │
│     58 │   is_package_available,                                                                                                                                                          │
│     59 )                                                                                                                                                                                  │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/utils/cache_management.py:38 in <module>                                                                                  │
│                                                                                                                                                                                           │
│    35 from lighteval.models.abstract_model import ModelConfig                                                                                                                             │
│    36 from lighteval.models.model_output import ModelResponse                                                                                                                             │
│    37 from lighteval.tasks.lighteval_task import LightevalTaskConfig                                                                                                                      │
│ ❱  38 from lighteval.tasks.registry import Registry                                                                                                                                       │
│    39 from lighteval.tasks.requests import Doc, SamplingMethod                                                                                                                            │
│    40 from lighteval.utils.utils import as_list                                                                                                                                           │
│    41                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/registry.py:37 in <module>                                                                                          │
│                                                                                                                                                                                           │
│    34 from types import ModuleType                                                                                                                                                        │
│    35                                                                                                                                                                                     │
│    36 import lighteval.tasks.default_tasks as default_tasks                                                                                                                               │
│ ❱  37 from lighteval.tasks.extended import AVAILABLE_EXTENDED_TASKS_MODULES                                                                                                               │
│    38 from lighteval.tasks.lighteval_task import LightevalTask, LightevalTaskConfig                                                                                                       │
│    39                                                                                                                                                                                     │
│    40                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/extended/__init__.py:25 in <module>                                                                                 │
│                                                                                                                                                                                           │
│   22                                                                                                                                                                                      │
│   23                                                                                                                                                                                      │
│   24 import lighteval.tasks.extended.hle.main as hle                                                                                                                                      │
│ ❱ 25 import lighteval.tasks.extended.ifbench.main as ifbench                                                                                                                              │
│   26 import lighteval.tasks.extended.ifeval.main as ifeval                                                                                                                                │
│   27 import lighteval.tasks.extended.lcb.main as lcb                                                                                                                                      │
│   28 import lighteval.tasks.extended.mix_eval.main as mix_eval                                                                                                                            │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/extended/ifbench/main.py:33 in <module>                                                                             │
│                                                                                                                                                                                           │
│    30 │   SampleLevelMetricGrouping,                                                                                                                                                      │
│    31 )                                                                                                                                                                                   │
│    32 from lighteval.models.model_output import ModelResponse                                                                                                                             │
│ ❱  33 from lighteval.tasks.extended.ifbench import evaluation_lib                                                                                                                         │
│    34 from lighteval.tasks.lighteval_task import LightevalTaskConfig                                                                                                                      │
│    35 from lighteval.tasks.requests import Doc, SamplingMethod                                                                                                                            │
│    36                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/extended/ifbench/evaluation_lib.py:23 in <module>                                                                   │
│                                                                                                                                                                                           │
│    20 import json                                                                                                                                                                         │
│    21 from typing import Dict, Optional, Union                                                                                                                                            │
│    22                                                                                                                                                                                     │
│ ❱  23 import lighteval.tasks.extended.ifbench.instructions_registry as instructions_registry                                                                                              │
│    24                                                                                                                                                                                     │
│    25                                                                                                                                                                                     │
│    26 @dataclasses.dataclass                                                                                                                                                              │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/extended/ifbench/instructions_registry.py:17 in <module>                                                            │
│                                                                                                                                                                                           │
│   14                                                                                                                                                                                      │
│   15 """Registry of all instructions."""                                                                                                                                                  │
│   16                                                                                                                                                                                      │
│ ❱ 17 import lighteval.tasks.extended.ifbench.instructions as instructions                                                                                                                 │
│   18                                                                                                                                                                                      │
│   19                                                                                                                                                                                      │
│   20 INSTRUCTION_DICT = {                                                                                                                                                                 │
│                                                                                                                                                                                           │
│ /home/mc/.venv-lighteval/lib/python3.10/site-packages/lighteval/tasks/extended/ifbench/instructions.py:26 in <module>                                                                     │
│                                                                                                                                                                                           │
│     23 import unicodedata                                                                                                                                                                 │
│     24 from collections import Counter                                                                                                                                                    │
│     25                                                                                                                                                                                    │
│ ❱   26 import emoji                                                                                                                                                                       │
│     27 import nltk                                                                                                                                                                        │
│     28                                                                                                                                                                                    │
│     29 from lighteval.utils.imports import is_package_available, requires                                                                                                                 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'emoji'

To Reproduce

Just run the basic example in a clean environment created with pip install lighteval.

lighteval accelerate \
  "model_name=openai-community/gpt2" \
  "leaderboard|truthfulqa:mc|0"

Expected behavior

I expected the truthfulqa:mc task to run successfully in a clean environment with only the base installation (pip install lighteval), without requiring any optional extended_tasks dependencies.

Would you like me to open a PR that moves emoji to the base dependencies?

Version info

$ pip freeze

absl-py==2.3.1
accelerate==1.10.1
aenum==3.1.15
aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.11.0
async-timeout==5.0.1
attrs==25.3.0
certifi==2025.8.3
chardet==5.2.0
charset-normalizer==3.4.3
click==8.3.0
colorama==0.4.6
colorlog==6.9.0
DataProperty==1.1.0
datasets==4.1.1
dill==0.4.0
exceptiongroup==1.3.0
filelock==3.19.1
frozenlist==1.7.0
fsspec==2025.9.0
gitdb==4.0.12
GitPython==3.1.45
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.35.3
idna==3.10
Jinja2==3.1.6
joblib==1.5.2
langcodes==3.5.0
language_data==1.3.0
latex2sympy2_extended==1.0.6
lighteval==0.11.0
lxml==6.0.2
marisa-trie==1.3.1
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mbstrdecoder==1.1.4
mdurl==0.1.2
mpmath==1.3.0
multidict==6.6.4
multiprocess==0.70.16
networkx==3.4.2
nltk==3.9.1
numpy==2.2.6
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-nccl-cu12==2.27.3
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvtx-cu12==12.8.90
packaging==25.0
pandas==2.3.3
pathvalidate==3.3.1
portalocker==3.2.0
propcache==0.3.2
protobuf==6.32.1
psutil==7.1.0
pyarrow==21.0.0
pycountry==24.6.1
pydantic==2.11.9
pydantic_core==2.33.2
Pygments==2.19.2
pytablewriter==1.2.1
python-dateutil==2.9.0.post0
pytz==2025.2
PyYAML==6.0.3
regex==2025.9.18
requests==2.32.5
rich==14.1.0
rouge-score==0.1.2
sacrebleu==2.5.1
safetensors==0.6.2
scikit-learn==1.7.2
scipy==1.15.3
sentencepiece==0.2.1
shellingham==1.5.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
sympy==1.14.0
tabledata==1.3.4
tabulate==0.9.0
tcolorpy==0.1.7
termcolor==2.3.0
threadpoolctl==3.6.0
tokenizers==0.22.1
torch==2.8.0
tqdm==4.67.1
transformers==4.56.2
triton==3.4.0
typepy==1.3.4
typer==0.19.2
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
urllib3==2.5.0
xxhash==3.6.0
yarl==1.20.1

[mcavdar]

similar case:

$ lighteval tasks list

[2025-10-03 08:26:08,812] [ WARNING]: You passed no task name. This should only occur if you are using the CLI to inspect tasks. (registry.py:152)
╭──────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────────────╮
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/main_tasks.py:78 in list                                                                                                      │
│                                                                                                                                                                                           │
│   75 │   """List all tasks"""                                                                                                                                                             │
│   76 │   from lighteval.tasks.registry import Registry                                                                                                                                    │
│   77 │                                                                                                                                                                                    │
│ ❱ 78 │   registry = Registry(custom_tasks=custom_tasks, load_community=True, load_extended=Tr                                                                                             │
│   79 │   registry.print_all_tasks(suites=suites)                                                                                                                                          │
│   80                                                                                                                                                                                      │
│   81                                                                                                                                                                                      │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/tasks/registry.py:166 in __init__                                                                                             │
│                                                                                                                                                                                           │
│   163 │   │   self._activate_loading_of_optional_suite()  # we dynamically set the loading par                                                                                            │
│   164 │   │                                                                                                                                                                               │
│   165 │   │   # We load all task to                                                                                                                                                       │
│ ❱ 166 │   │   self._task_registry = self._load_full_registry()                                                                                                                            │
│   167 │   │                                                                                                                                                                               │
│   168 │   │   self.task_to_configs = self._update_task_configs()                                                                                                                          │
│   169                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/tasks/registry.py:253 in _load_full_registry                                                                                  │
│                                                                                                                                                                                           │
│   250 │   │                                                                                                                                                                               │
│   251 │   │   # Need to load multilingual tasks                                                                                                                                           │
│   252 │   │   if self._load_multilingual:                                                                                                                                                 │
│ ❱ 253 │   │   │   import lighteval.tasks.multilingual.tasks as multilingual_tasks                                                                                                         │
│   254 │   │   │                                                                                                                                                                           │
│   255 │   │   │   custom_tasks_module.append(multilingual_tasks)                                                                                                                          │
│   256                                                                                                                                                                                     │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/tasks/multilingual/tasks.py:820 in <module>                                                                                   │
│                                                                                                                                                                                           │
│    817                                                                                                                                                                                    │
│    818 # XQuAD: Cross-lingual Question Answering Dataset, extending SQuAD to 11 languages.                                                                                                │
│    819 # https://arxiv.org/abs/1910.11856                                                                                                                                                 │
│ ❱  820 xquad_tasks = [                                                                                                                                                                    │
│    821 │   LightevalTaskConfig(                                                                                                                                                           │
│    822 │   │   name=f"xquad_{language.value}",                                                                                                                                            │
│    823 │   │   prompt_function=get_qa_prompt_function(                                                                                                                                    │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/tasks/multilingual/tasks.py:839 in <listcomp>                                                                                 │
│                                                                                                                                                                                           │
│    836 │   │   generation_size=400,                                                                                                                                                       │
│    837 │   │   stop_sequence=("\n",),                                                                                                                                                     │
│    838 │   │   metrics=(                                                                                                                                                                  │
│ ❱  839 │   │   │   MultilingualQuasiExactMatchMetric(language, "prefix"),                                                                                                                 │
│    840 │   │   │   MultilingualQuasiF1ScoreMetric(language),                                                                                                                              │
│    841 │   │   ),                                                                                                                                                                         │
│    842 │   )                                                                                                                                                                              │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/metrics/dynamic_metrics.py:144 in __init__                                                                                    │
│                                                                                                                                                                                           │
│   141 │   │   super().__init__(                                                                                                                                                           │
│   142 │   │   │   metric_name=f"exact_match_{language.value}_{match_type}",                                                                                                               │
│   143 │   │   │   sample_level_fn=ExactMatches(                                                                                                                                           │
│ ❱ 144 │   │   │   │   normalize_gold=get_multilingual_normalizer(language),                                                                                                               │
│   145 │   │   │   │   normalize_pred=get_multilingual_normalizer(language),                                                                                                               │
│   146 │   │   │   │   aggregation_function=aggregation_function,                                                                                                                          │
│   147 │   │   │   │   type_exact_match=match_type,                                                                                                                                        │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/utils/imports.py:211 in wrapper                                                                                               │
│                                                                                                                                                                                           │
│   208 │   │   │   @functools.wraps(_object)                                                                                                                                               │
│   209 │   │   │   def wrapper(*args, **kwargs):                                                                                                                                           │
│   210 │   │   │   │   for backend in _object._backends:                                                                                                                                   │
│ ❱ 211 │   │   │   │   │   raise_if_package_not_available(backend, object_name=_object.__name__                                                                                            │
│   212 │   │   │   │   return _object(*args, **kwargs)                                                                                                                                     │
│   213 │   │   │                                                                                                                                                                           │
│   214 │   │   │   return wrapper                                                                                                                                                          │
│                                                                                                                                                                                           │
│ /home/mc/.venv-huggo/lib/python3.10/site-packages/lighteval/utils/imports.py:115 in raise_if_package_not_available                                                                        │
│                                                                                                                                                                                           │
│   112 │   prefix = "You" if object_name is None else f"Through the use of {object_name}, you"                                                                                             │
│   113 │                                                                                                                                                                                   │
│   114 │   if package == Extra.MULTILINGUAL and ((language is not None) or (not is_multilingual                                                                                            │
│ ❱ 115 │   │   raise ImportError(prefix + not_installed_error_message(package)[3:])                                                                                                        │
│   116 │                                                                                                                                                                                   │
│   117 │   if not is_package_available(package):                                                                                                                                           │
│   118 │   │   raise ImportError(prefix + not_installed_error_message(package)[3:])                                                                                                        │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: Through the use of get_multilingual_normalizer, you are trying to run an evaluation requiring multilingual capabilities. Please install the required extra: `pip install
lighteval[multilingual]`

[mcavdar]

I just wanted to note that Python is introducing PEP 810 – Explicit Lazy Imports, which allows certain imports to be deferred until they’re actually used.

It’s worth keeping an eye on PEP 810, but for now, we’ll likely need to explicitly install the missing dependency or adjust the code until lazy imports are available in stable Python releases.

[clefourrier]

Ooooh extremely cool! Thanks a lot! cc @NathanHB for viz