Skip to content

Comments

fix: run lmms-eval --tasks list & lmms-eval --tasks list_with_num command correctly#46

Open
akawincent wants to merge 2 commits intovision-x-nyu:mainfrom
akawincent:for-pr
Open

fix: run lmms-eval --tasks list & lmms-eval --tasks list_with_num command correctly#46
akawincent wants to merge 2 commits intovision-x-nyu:mainfrom
akawincent:for-pr

Conversation

@akawincent
Copy link

@akawincent akawincent commented Jan 22, 2026

This PR restores the lmms-eval --tasks list and lmms-eval --tasks list_with_num command, fundamental features of lmms-eval that ensures users can query available tasks and their corresponding question counts.. Maintaining this feature provides better compatibility for future benchmark variants, such as vsi-bench-debias.

lmms-eval --tasks list

Before: Fail to display the task list correctly.

This script was launched with accelerate
2026-01-23 00:20:03.458 | INFO     | lmms_eval.__main__:cli_evaluate:308 - Verbosity set to INFO
2026-01-23 00:20:03.558 | INFO     | lmms_eval.__main__:cli_evaluate_single:392 - Evaluation tracker args: {}
2026-01-23 00:20:03.791 | INFO     | lmms_eval.__main__:cli_evaluate_single:428 - Available Tasks:
 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 - 

 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 -  
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - -
 - .
 - /
 - /
 - /
 - C
 - C
 - G
 - L
 - L
 - O
 - T
 - T
 - T
 - _
 - _
 - a
 - a
 - a
 - a
 - a
 - a
 - a
 - a
 - b
 - b
 - b
 - c
 - c
 - c
 - c
 - c
 - e
 - e
 - e
 - e
 - e
 - e
 - e
 - e
 - f
 - f
 - g
 - g
 - g
 - g
 - h
 - h
 - h
 - i
 - i
 - i
 - i
 - i
 - i
 - i
 - i
 - k
 - k
 - l
 - l
 - l
 - l
 - m
 - m
 - m
 - n
 - n
 - n
 - n
 - n
 - n
 - n
 - n
 - n
 - o
 - o
 - o
 - o
 - o
 - o
 - o
 - p
 - p
 - p
 - r
 - r
 - s
 - s
 - s
 - s
 - s
 - s
 - s
 - t
 - t
 - t
 - t
 - t
 - t
 - t
 - u
 - u
 - u
 - u
 - v
 - v
 - v
 - v
 - y
 - y
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |
 - |

After: Correctly output all registered tasks.

This script was launched with accelerate
2026-01-23 00:28:20.934 | INFO     | lmms_eval.__main__:cli_evaluate:308 - Verbosity set to INFO
2026-01-23 00:28:21.031 | INFO     | lmms_eval.__main__:cli_evaluate_single:392 - Evaluation tracker args: {}
2026-01-23 00:28:21.036 | INFO     | lmms_eval.__main__:cli_evaluate_single:428 - Available Tasks:
 - vsibench

lmms-eval --tasks list_with_num

Before: No results

This script was launched with accelerate
2026-01-23 01:14:11.348 | INFO     | lmms_eval.__main__:cli_evaluate:308 - Verbosity set to INFO
2026-01-23 01:14:11.445 | INFO     | lmms_eval.__main__:cli_evaluate_single:392 - Evaluation tracker args: {}
2026-01-23 01:14:11.448 | INFO     | lmms_eval.__main__:cli_evaluate_single:443 - 
======================================================================

        You are trying to check all the numbers in each task.
        This action will download the complete dataset.
        If the results are not clear initially, call this again.

======================================================================

After: Correctly output all registered tasks with their numbers.

This script was launched with accelerate
2026-01-23 01:12:03.028 | INFO     | lmms_eval.__main__:cli_evaluate:308 - Verbosity set to INFO
2026-01-23 01:12:03.130 | INFO     | lmms_eval.__main__:cli_evaluate_single:392 - Evaluation tracker args: {}
2026-01-23 01:12:03.135 | INFO     | lmms_eval.__main__:cli_evaluate_single:443 - 
======================================================================

        You are trying to check all the numbers in each task.
        This action will download the complete dataset.
        If the results are not clear initially, call this again.

======================================================================
Fetching 10 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 7913.78it/s]
2026-01-23 01:12:06.618 | INFO     | lmms_eval.__main__:cli_evaluate_single:452 - 
Task : vsibench
 - #num : 5130

@akawincent akawincent changed the title fix: run lmms-eval --tasks list command correctly fix: run lmms-eval --tasks list & lmms-eval --tasks list_with_num command correctly Jan 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant