Format of `fail_to_pass` in Hugging Face is not valid JSON

[jtmcg]

While trying to extract the test cases used for assigning a successful or failed eval, we noticed the test cases in fail_to_pass are not valid JSON:

example from benchmark instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan:

["test/database.js | Test database test/database/keys.js::Key methods should return multiple keys and null if key doesn't exist", 'test/database.js | Test database test/database/keys.js::Key methods should return empty array if keys is empty array or falsy', 'test/user/emails.js | email confirmation (library methods) canSendValidation should return true if it has been long enough to re-send confirmation']

This means that when trying to load those test cases we are getting an error:

import json
from datasets import load_dataset
ds = load_dataset("ScaleAI/SWE-bench_Pro")
json.loads(ds['test'][0]['fail_to_pass'])

# json.decoder.JSONDecodeError: Expecting value: line 1 column 131 (char 130)

[jeff-da]

Can you try to load the strings using eval instead of json?

[colinwm]

Yes, it works with eval. But isn't that a bit strange? Why would the format differ compared to pass_to_pass? Also, it makes it harder to parse in languages other than python.

[jeff-da]

Hey @colinwm , sorry about the extra trouble. For why f2p and p2p are different, I wrote some parsing script for f2p after the human validation step. But it is a bit strange, since I did save the column as a JSON, looking into this.