Skip to content

NVIDIA NeMo Evaluator 0.1.72

Choose a tag to compare

@ko3n1g ko3n1g released this 28 Jan 12:31
193483d
fix: restore support for running tasks not listed in FDF (#667)

We have improved our validation in the spirit of failing early. However,
this lead to unwanted side effect - we've lost support for running tasks
not listed in FDF with `harness.task` syntax. Calling evaluation with
this syntax was resulting in
`nemo_evaluator.core.utils.MisconfigurationError: Unknown evaluation
xxx`

It stopped working because:
* we run validation (everything passes here)
* then we prepare the config, extracting `task` from `harness.task` and
using in as evaluation `type`
* we run 2nd validation and it fails because we no longer use
`harness.task` syntax and there's no evaluation called
`task` in FDF

This PR uses `harness.task` as `type` to make sure it's always valid +
adds test verifying custom task support. It also removes one redundant
validation

---------

Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>