[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it

[colinzuo]

Describe the bug

Specify two tasks as below:
helm|boolq|0
helm|boolq:contrastset|0
The current logic will assume "helm|boolq" is a group and will add "helm|boolq:contrastset" again, then "helm|boolq:contrastset" will appear twice, later at SampleCache:_get_task_hash sorted is used to sort these two, and
an error "<" is not supported on LightevalTaskConfig is raised

To Reproduce

Specify two tasks as below in yaml:
helm|boolq|0
helm|boolq:contrastset|0

Expected behavior

Run should succeed

Version info

0.11.0

[clefourrier]

Oh cool edge case, good catch! Here we would actually rename boolq to boolq:default so that a task does not also carry a group name.

[colinzuo]

How about sorted(self.registry.task_to_configs[f"{task_suite}|{task_name}"]), currently comparison operator is not defined for LightevalTaskConfig, so this place will fail if more than one config is met, any thought?