Skip to content

Commit e1a7a39

Browse files
authored
change multimodal check in evaluate (#3013)
changed multimodal check from strict equality
1 parent 29ea683 commit e1a7a39

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

lm_eval/evaluator.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -483,7 +483,7 @@ def evaluate(
483483
for task_output in eval_tasks:
484484
task: Task = task_output.task
485485

486-
if getattr(lm, "MULTIMODAL", False) != getattr(task, "MULTIMODAL", False):
486+
if getattr(task, "MULTIMODAL", False) and not getattr(lm, "MULTIMODAL", False):
487487
incompatible_tasks.append(task_output.task_name)
488488
elif getattr(task, "UNSAFE_CODE", False) and not confirm_run_unsafe_code:
489489
raise ValueError(

0 commit comments

Comments
 (0)