fix tasks list #906
fix tasks list #906
Conversation
|
++ @clefourrier |
|
@Winlere check this out if it works for you ? |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@NathanHB i've added the commit with your suggestions. |
| tuple[list[str], dict[str, list[tuple[int, bool]]]]: A tuple containing: | ||
| - A sorted list of unique task names in the format "suite|task". | ||
| - A dictionary mapping each task name to a list of tuples representing the few_shot and truncate_few_shots values. |
|
need to fix styling and remove the docstring that changed and will merge :) |
|
One last nit, maybe add a kwarg like --with-community or --with-multilingual to list all tasks, so that people not wanting to get 74K lines of output can look at the other tasks |
alielfilali01
force-pushed
the
alielfilali01-patch-1-fixTasksList
branch
from
c7f6f4e to
34eba92
Compare
|
@clefourrier , multilingual is always part of the |
|
multilingual are not part of the core tasks of lighteval, as you need to install a specific set of dependencies for them (and therefore you should not change the pyproject.toml but test if the user has installed these deps when requesting multilingual tasks) |
|
I like your idea of |
alielfilali01
force-pushed
the
alielfilali01-patch-1-fixTasksList
branch
from
ddf1b03 to
d0cd4c9
Compare
refix the tasks list command
|
need me to reopen it? |
|
Sorry guys, i've pushed from the wrong local directory which cuased an empty commit ! |
|
np ^^ |
feat(tasks): add suite filtering and fix community task discovery - Revert the community task path fix - Add --suites parameter to lighteval tasks list - Default to core suites only to prevent overwhelming output - Add dependency checking for multilingual tasks
feat(tasks): add suite filtering and fix community task discovery - Revert the community task path fix - Add --suites parameter to lighteval tasks list - Default to core suites only to prevent overwhelming output - Add dependency checking for multilingual tasks
feat(tasks): Enhance task discovery and display. Bug mentioned here: #903
This PR improves the task registry to ensure all available task suites, including community, custom, test suites ... are correctly discovered and displayed in the
lighteval tasks listcommand.The key changes include:
Dynamic Community Task Loading: the new function (
load_community_tasks) is to dynamically scan thecommunity_tasksdirectory and load any valid task modules. This process is resilient, as it gracefully handles and logs errors for any malformed task files without interrupting the loading of others.Lazy Loading for Multilingual Tasks: The import for multilingual tasks is now deferred until the task registry is accessed. This prevents crashes when optional dependencies (like
langcodes) are not installed and instead logs a helpful warning.Comprehensive Suite Display: The
print_all_tasksfunction has been updated to guarantee that allDEFAULT_SUITES(includingcustomandtest) are always visible in the output, even if they contain no tasks. A "(no tasks in this suite)" message is now shown for empty suites to improve clarity.These changes provide a more robust and complete discovery mechanism, giving users a reliable view of all available tasks in
lighteval.Before these changes, the command
lighteval tasks listused to return about 1500 lines (tasks).After these changes, the command
lighteval tasks listnow returns more than 74200 lines (tasks).