Skip to content

Commit 8ada805

Browse files
authored
[Bug] fix a bug in post processing stage of ScienceQA. (#723)
* fix a bug in sqa_process_results * reformat code
1 parent 2dad280 commit 8ada805

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

lmms_eval/tasks/scienceqa/utils.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,11 @@ def sqa_doc_to_target(doc):
3434
def sqa_process_results(doc, results):
3535
# I know this is weird, but it's how llava parse it.
3636
target = sqa_doc_to_target(doc).strip().lower()
37-
pred = results[0].strip().lower()
38-
if pred == target:
37+
pred = results[0].strip()
38+
if pred.lower() == target:
3939
return {"exact_match": 1.0}
4040
# pattern: ^[A-Z]\. .*
4141
if len(pred) >= 2 and pred[0].isupper() and pred[1] == ".":
42-
result = 1.0 if pred[0] == target else 0.0
42+
result = 1.0 if pred[0].lower() == target else 0.0
4343
return {"exact_match": result}
4444
return {"exact_match": 0.0}

0 commit comments

Comments
 (0)