You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix: restore support for running tasks not listed in FDF (#667)
We have improved our validation in the spirit of failing early. However,
this lead to unwanted side effect - we've lost support for running tasks
not listed in FDF with `harness.task` syntax. Calling evaluation with
this syntax was resulting in
`nemo_evaluator.core.utils.MisconfigurationError: Unknown evaluation
xxx`
It stopped working because:
* we run validation (everything passes here)
* then we prepare the config, extracting `task` from `harness.task` and
using in as evaluation `type`
* we run 2nd validation and it fails because we no longer use
`harness.task` syntax and there's no evaluation called
`task` in FDF
This PR uses `harness.task` as `type` to make sure it's always valid +
adds test verifying custom task support. It also removes one redundant
validation
---------
Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>