feat: vlm bench warehouse tasks#668

Open

MagdalenaKotynia wants to merge 18 commits intomainfrom

mk/feat/vlm-bench-warehouse-tasks

Commits on Aug 18, 2025

feat: interfaces for quantity and multiple choice tasks
MagdalenaKotynia
committed
feat: created new tasks with warehouse simulation images
MagdalenaKotynia
committed
feat: added vlm bench config to test many models
MagdalenaKotynia
committed
chore: moved old image files from vlm benchmark to lfs
MagdalenaKotynia
committed
refactor: extracted common logic for vlm tasks inputs and answers to base ImageReasoningTaskInput and ImageReasoningAnswer classes
MagdalenaKotynia
committed
docs: added info about increasing event size for Langfuse tracing of vlm benchmark
MagdalenaKotynia
committed
feat: added merging results summaries across all repeats and all models
MagdalenaKotynia
committed
fix: fixed typing after refactor extracting common logic for vlm tasks inputs and answers
MagdalenaKotynia
committed
refactor: reorganized tasks order by images
MagdalenaKotynia
committed
chore: removed unused comment
MagdalenaKotynia
committed
refactor: created separate structure for storing summary for the model across all tasks and repeats
MagdalenaKotynia
committed
fix: fixed validation of parsed llm output
MagdalenaKotynia
committed
refactor: added task id, refactor of storing task_input variables
MagdalenaKotynia
committed
fix: aggregate tasks repeats results by the task id
MagdalenaKotynia
committed
refactor: renamed csv storing model summary
MagdalenaKotynia
committed
feat: added std success rate and std time for model across all repeats
MagdalenaKotynia
committed

Commits on Aug 27, 2025