feat: add MultihopSpatial task#1402
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds the new MultihopSpatial benchmark task to lmms_eval, enabling evaluation of multi-hop spatial reasoning with both multiple-choice answering and bounding-box grounding metrics.
Changes:
- Introduces
multihopspatialtask configuration (HF dataset wiring + metric definitions). - Adds task-specific prompting, MCQ parsing, bbox parsing, IoU scoring, and metric aggregation utilities.
- Updates the “Spatial & Grounding Tasks” docs list to include MultihopSpatial.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
lmms_eval/tasks/multihopspatial/utils.py |
Implements prompt formatting, prediction parsing (MCQ + bbox), IoU computation, and metric aggregation for the task. |
lmms_eval/tasks/multihopspatial/multihopspatial.yaml |
Registers the task against the HF dataset and hooks the task utilities + metrics into lmms-eval. |
docs/advanced/current_tasks.md |
Adds MultihopSpatial to the documented list of spatial/grounding tasks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| m = re.search( | ||
| r'"bbox_2d"\s*:\s*\[\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*\]', | ||
| response_text, | ||
| ) | ||
| if m is None: | ||
| m = re.search( | ||
| r"Bounding Box:\s*\[\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*\]", | ||
| response_text, | ||
| re.IGNORECASE, | ||
| ) | ||
| if m is not None: | ||
| bbox = [float(m.group(i)) for i in range(1, 5)] | ||
| else: | ||
| found = re.findall(r"\[\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*,\s*([\d.]+)\s*\]", response_text) | ||
| if found: | ||
| bbox = [float(v) for v in found[0]] | ||
|
|
There was a problem hiding this comment.
Good catch — fixed. The coordinate regex is now a well-formed float pattern (\d*\.?\d+) and the float() conversion is wrapped in try/except, so malformed model output (e.g. 1.2.3, ..) returns None instead of raising. Normal parsing (including 0-1000 auto-rescale and shorthand decimals like .5) is unchanged, so the reported numbers are unaffected.
| if union <= 0: | ||
| return 0.0 | ||
| return round(inter / union, 4) |
There was a problem hiding this comment.
Kept intentionally. The 4-decimal rounding mirrors the reference implementation these reported numbers were produced with, so the task reproduces the paper's Acc@50IoU / Avg IoU exactly. Dropping it would make the lmms-eval numbers diverge slightly from the published ones. The borderline-flip case requires an IoU in [0.49995, 0.5), which is negligible in practice, so I've left it as-is for reproducibility.
MultihopSpatial evaluates multi-hop compositional spatial reasoning with visual grounding. Each sample is a multiple-choice question paired with a bounding box that localizes the answer, so a model must both pick the correct choice and localize it - exposing the "lucky guess" gap where MCQ accuracy is high but grounding is poor. Metrics: mcq_acc, avg_iou (over MCQ-correct samples), and acc50iou (both MCQ-correct and IoU >= 0.5). A single prompt is used for every model, asking for a normalized-0-1 xyxy box; the parser normalizes 0-1000 outputs uniformly so no model is disadvantaged by its native coordinate scale. Dataset: https://huggingface.co/datasets/etri-vilab/MultihopSpatial
d023388 to
780da78
Compare
Summary
mcq_acc,avg_iou(over MCQ-correct samples), andacc@50iou(both MCQ-correct and IoU ≥ 0.5).In scope
lmms_eval/tasks/multihopspatial/(multihopspatial.yaml,utils.py).docs/advanced/current_tasks.mdunder "7. Spatial & Grounding Tasks".etri-vilab/MultihopSpatial(benchmarkconfig, image-embedded parquet).Out of scope
Validation
python -m lmms_eval --model vllm --model_args model=etri-vilab/MultiHopSpatial-Qwen3-VL-4B-Instruct --tasks multihopspatial --batch_size 128| sample size:N=4500| key metrics:mcq_acc=63.87, acc50iou=50.76, avg_iou=68.95| result:passpython -m lmms_eval --model vllm --model_args model=Qwen/Qwen3-VL-4B-Instruct --tasks multihopspatial --batch_size 128| sample size:N=4500| key metrics:mcq_acc=37.91, acc50iou=25.87, avg_iou=61.34| result:passqwen3_vl(transformers) backend;black --line-length 240andisortpass.Risk / Compatibility
Type of Change
Uniform evaluation protocol
A single prompt is used for every model, asking for a normalized-0–1
xyxybox as{"bbox_2d": [...]}. To keep comparison fair, the parser normalizes coordinate scale uniformly (a 0–1000 output is divided by 1000), so no model is disadvantaged by its native scale; axis order is taken asxyxyas prompted (no per-model accommodation).