Fix caching logic#994

jxmorris12 wants to merge 5 commits intohuggingface:mainhuggingface/lighteval:mainfrom

jxmorris12:mainjxmorris12/lighteval:mainCopy head branch name to clipboard