Add evaluator class #4
Description
Up for discussion.
A stateful class could help enabling the following:
- Track evaluations through time in a simple way
- Store multiple-metric evals in a single object (users may want to eval a model with multiple metrics and consolidate all of these in a single artifact, which would be easier to produce from a single object)
- Focus on reproducibility: there should be a simple mechanism that enables automatic logging of everything that is done to/with a marked instance
- Scheduled support (i.e. cron jobs)? Not 100% sold on this one, it may be better to implement it within mdb proper