Conversation
marta-sd
left a comment
marta-sd
left a comment
There was a problem hiding this comment.
Looks really good! I added some minor comments
README.md
Outdated
| ## 🚀 Features | ||
|
|
||
| - **Multi-Backend Deployment**: Support for both PyTriton and Ray Serve deployment backends | ||
| - **Comprehensive Evaluation**: Integration with NVIDIA Evals Factory for standardized benchmark evaluation |
There was a problem hiding this comment.
So at this point, the user is not introduced to what Evals Factory is, and how it relates to evaluation being comprehensive.
There was a problem hiding this comment.
Addressed this above in 56873d5, by adding some details about what NVIDIA Evals Factory is at the beginning. WDYT @shashank3959 ?
There was a problem hiding this comment.
Not relevant anymore since we decided to not use the term NVIDIA Evals Factory in the initial sections. Replaced with state-of-the-art evaluation harnesses under features
marta-sd
left a comment
marta-sd
left a comment
There was a problem hiding this comment.
I verified with the management that "NVIDIA Eval Factory" is the correct name. I think I suggested changes for all occurrences. Sorry for the back and forth.
|
/ok to test 28730d4 |
|
/ok to test d8851a5 |
Co-authored-by: Wenwen Gao <94138584+snowmanwwg@users.noreply.github.com>
Co-authored-by: Wenwen Gao <94138584+snowmanwwg@users.noreply.github.com>
Co-authored-by: Wenwen Gao <94138584+snowmanwwg@users.noreply.github.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
| [](https://github.com/NVIDIA-NeMo/Eval/blob/main/pyproject.toml) | ||
| [](https://github.com/NVIDIA-NeMo/) | ||
|
|
||
| [Documentation](https://nemo-framework-documentation.gitlab-master-pages.nvidia.com/eval-build/) | [Examples](#-usage-examples) | [Contributing](https://github.com/NVIDIA-NeMo/Eval/blob/main/CONTRIBUTING.md) |
There was a problem hiding this comment.
Documentation link does not work yet since the docs PR #4 is not merged. Once merged and the build is added in gitlab repo, this link should work. Will verify the link once this is done.
| ```bash | ||
| # Run all tests | ||
| pytest tests | ||
|
|
||
| # Run unit tests only | ||
| pytest tests/unit_tests/ | ||
|
|
||
| # Run functional tests only | ||
| pytest tests/functional_tests/ | ||
|
|
||
| # Run with coverage | ||
| pytest --cov=nemo_eval tests | ||
| ``` | ||
|
|
||
| #### Test Scripts | ||
|
|
There was a problem hiding this comment.
Let's remove this. I think it's better to point to the scripts that we use in CI
There was a problem hiding this comment.
Yeah we can remove this. Can you point me to the CI scripts you are refering to here ? Thanks!
There was a problem hiding this comment.
@ko3n1g could you please help modify this in a follow up ? Thanks!
|
/ok to test b83bbce |
Signed-off-by: Abhishree <abhishreetm@gmail.com>
|
Fast merging since its README. linting, copyright and DCO checks have passed. |
Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Wenwen Gao <94138584+snowmanwwg@users.noreply.github.com>
6 participants
Add README.md