[RFC Request] How to do eval?

[felipemello1]

HF Smol models use https://github.com/huggingface/lighteval
Some internal tooling uses https://github.com/UKGovernmentBEIS/inspect_ai

Tracking loss functions and rewards are not enough. Ideally, we should run the model on eval datasets and check results every N steps.

e.g. lets say your training data is wikipedia, but your eval is "humanity's last exam". Every N steps we want to check the score on "humanity's last exam".

[felipemello1]

From Smol training playbook: https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook#automated-evaluations

[allenwang28]

agree with the premise, my 2c is that LightEval looks nice and simple to implement. Why not go ahead and use it until it stops working for us?