Skip to content

[BUG] Cache management failed for tasks using MetricGrouping #1023

@colinzuo

Description

@colinzuo

Describe the bug

SampleCache:_get_task_hash uses LightevalTaskConfig:str to compute task hash, but LightevalTaskConfig:str
doesn't handle tasks using MetricGrouping correctly, for MetricGrouping field corpus_level_fn is a dict, so another level of loop is needed.

To Reproduce

Cache doesn't work for "leaderboard|truthfulqa:mc|0"

Expected behavior

Cache should work

Version info

0.11.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions