Search results

[BUG] fail to execute task harness|bbh:movie_recommendation|0

bug

Status: Open.

#1030 In huggingface/lighteval;

· colinzuo opened on Oct 27, 2025

[BUG] Python API doesn't seem to work with lighteval|gsm8k|<few-shot-number>

bug

Status: Open.

#1029 In huggingface/lighteval;

· Saravana-Ace opened on Oct 24, 2025

How to evaluate MMLU-Pro

Status: Open.

#1028 In huggingface/lighteval;

· qhz991029 opened on Oct 24, 2025

[BUG] Cache management failed for tasks using sample_params to update function attrs

bug

Status: Open.

#1026 In huggingface/lighteval;

· colinzuo opened on Oct 22, 2025

[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it

bug

Status: Open.

#1024 In huggingface/lighteval;

· colinzuo opened on Oct 21, 2025

[BUG] Cache management failed for tasks using MetricGrouping

bug

Status: Open.

#1023 In huggingface/lighteval;

· colinzuo opened on Oct 21, 2025

Vision Language Model Support?

Status: Open.

#1020 In huggingface/lighteval;

· ByungKwanLee opened on Oct 15, 2025

[EVAL] MultiChallenge

Status: Open.

#1019 In huggingface/lighteval;

· lewtun opened on Oct 15, 2025

[BUG] MixEval crashes with <code>KeyError: TaskID(task_name='extended|mixeval_hard:multichoice|0', task_hash='5029516ccc122911', sampling_method=&lt;SamplingMethod.GENERATIVE: 'GENERATIVE'&gt;)</code>

bug

Status: Open.

#1005 In huggingface/lighteval;

· lewtun opened on Oct 3, 2025

[BUG] truthfulqa requires extended_tasks dependencies because registry eagerly imports all extended tasks

bug

Status: Open.

#1004 In huggingface/lighteval;

· mcavdar opened on Oct 3, 2025

[FT] Upgrade the VLLM dependency to 0.10.2+

Status: Open.

#1002 In huggingface/lighteval;

· JIElite opened on Oct 2, 2025

RULER benchmark not found

Status: Open.

#1001 In huggingface/lighteval;

· Roadsong opened on Sep 30, 2025