The command-line argument "batch_size" being set to “auto” is applicable to all models?

[CodeTeamster]

When I use lmms_eval in the following manner, the code throw exception that 'ValueError: invalid literal for int() with base 10: 'auto'.

accelerate launch $CONFIG_FILE_ARG
-m lmms_eval
--model qwen2_5_vl
--model_args pretrained="${PRETRAINED_MODEL_PATH},min_pixels=${MIN_PIXELS},max_pixels=${MAX_PIXELS},attn_implementation=flash_attention_2,interleave_visuals=False"
--tasks $TASKS
--verbosity WARNING
--batch_size auto
--log_samples
--log_samples_suffix $RUN_NAME
--output_path $LOG_DIR/$RUN_NAME

[Luodian]

The batch_size=auto feature isn't implemented for most models in lmms-eval. Looking at the codebase, the models under lmms_eval/models/simple/ (including qwen2_5_vl) expect an integer batch size.

Use an explicit integer instead:

--batch_size 8

Start with a smaller value and increase until you hit OOM, then back off.

[CodeTeamster]

The batch_size=auto feature isn't implemented for most models in lmms-eval. Looking at the codebase, the models under lmms_eval/models/simple/ (including qwen2_5_vl) expect an integer batch size.

Use an explicit integer instead:

--batch_size 8
Start with a smaller value and increase until you hit OOM, then back off.

Thank you for your reply. At the same time, I think some additional explanations regarding this parameter can be added to lmms-eval/docs/commands.md

[Luodian]

Good point. The docs should clarify that batch_size=auto isn't implemented for most models. A PR updating docs/commands.md to note this would be helpful.