[BUG] Python API doesn't seem to work with lighteval|gsm8k|<few-shot-number>

[Saravana-Ace]

Describe the bug

Lighteval's python API doesn't seem to work with lighteval|gsm8k|. It looks like LightEval is looking for the GSM8K dataset under lighteval/openai/gsm8k instead of using the Hugging Face dataset loader.

To Reproduce

Run the following code and you should get the following error:
FileNotFoundError: Couldn't find any data file at /lighteval/openai/gsm8k

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.transformers.transformers_model import (
    TransformersModel,
    TransformersModelConfig,
)
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


def load_model():
    base_model = "Qwen/Qwen2.5-7B"
    tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
    tokenizer.padding_side = "left"
    
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token_id = tokenizer.eos_token_id

    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        device_map="auto",
        torch_dtype=torch.bfloat16,
        trust_remote_code=True,
    )

    return model, tokenizer


def main():
    model, tokenizer = load_model()

    # Setup lighteval
    evaluation_tracker = EvaluationTracker(
        output_dir="./eval_results",
        save_details=True,
        push_to_hub=False,
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.NONE,
        custom_tasks_directory=None,
        max_samples=10,
    )

    config = TransformersModelConfig(
        model_name="Qwen/Qwen2.5-7B",
        batch_size=4,
    )

    lighteval_model = TransformersModel.from_model(model, config)

    # This triggers the FileNotFoundError
    pipeline = Pipeline(
        model=lighteval_model,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        tasks="lighteval|gsm8k|0",
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()


if __name__ == "__main__":
    main()

Expected behavior

You should see this error:
FileNotFoundError: Couldn't find any data file at /lighteval/openai/gsm8k

Version info

• OS: Linux (x86_64, kernel 5.14.0-503.19.1.el9_5.x86_64)
• LightEval version: 0.11.0
• Python version: 3.13 (conda environment)
• Conda: yes (Anaconda/Miniconda-based setup)
• Install method: pip install lighteval
• Location: site-packages within conda env