From e79b74342e82b98a55303a7ebeea3e2ccea7dd5f Mon Sep 17 00:00:00 2001 From: Yifan Mai Date: Wed, 25 Feb 2026 11:13:15 -0800 Subject: [PATCH] Add question_type=multiple-choice to VHELM example in docs --- docs/vhelm.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/vhelm.md b/docs/vhelm.md index 59fb31fdaf4..fc809c1bc83 100644 --- a/docs/vhelm.md +++ b/docs/vhelm.md @@ -24,11 +24,14 @@ pip install "crfm-helm[vlm]" The following is an example of evaluating `openai/gpt-4o-mini-2024-07-18` on 10 instance from the Accounting subset of MMMU. ```sh +# Set OpenAI API key +export OPENAI_API_KEY=your_api_key + # Download schema_vhelm.yaml wget https://raw.githubusercontent.com/stanford-crfm/helm/refs/heads/main/src/helm/benchmark/static/schema_vhelm.yaml # Run benchmark -helm-run --run-entries mmmu:subject=Accounting,model=openai/gpt-4o-mini-2024-07-18 --suite my-vhelm-suite --max-eval-instances 10 +helm-run --run-entries mmmu:subject=Accounting,question_type=multiple-choice,model=openai/gpt-4o-mini-2024-07-18 --suite my-vhelm-suite --max-eval-instances 10 # Summarize benchmark results helm-summarize --suite my-vhelm-suite --schema-path schema_vhelm.yaml