Skip to content

Pull requests: allenai/olmo-eval

New pull request New
39 Open 264 Closed
39 Open 264 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

ifeval_ood: read the public IFBench_test dataset
#326 opened Aug 25, 2026 by abhishekraok Contributor Loading…
Multi-image benchmarks + multimodal docs (5/5)
#324 opened Aug 25, 2026 by jason718 Loading…
6 tasks done
Image-QA benchmarks (4/5)
#323 opened Aug 25, 2026 by jason718 Loading…
5 tasks done
Pointing + counting benchmarks (3/5)
#322 opened Aug 25, 2026 by jason718 Loading…
5 tasks done
Add MMLU chain-of-thought tasks (post-training regime)
#321 opened Aug 25, 2026 by abhishekraok Contributor • Draft
olmo_core: accept uncapped max_tokens in the request trace
#319 opened Aug 25, 2026 by abhishekraok Contributor Loading…
DeepScholar-Bench as a first-class task on the shared paper-search agent scaffold
#318 opened Aug 25, 2026 by yilunzhao Contributor • Draft
Vision task framework + captioning benchmarks (2/5)
#317 opened Aug 25, 2026 by jason718 Loading…
5 tasks done
Multimodal eval (1/5): image requests, HF VLM path, OLMo-core VLM provider
#316 opened Aug 25, 2026 by jason718 Loading…
5 tasks done
@jason718
IFEval scorer: score unverifiable instructions as not followed
#314 opened Aug 23, 2026 by abhishekraok Contributor Loading…
5
[dup of #303] Exclude Torch companion packages and nvidia-cutlass-dsl from CUDA constraints
#312 opened Aug 23, 2026 by abhishekraok Contributor Loading…
3
Add GPQA 0-shot CoT tasks (post-training regime)
#309 opened Aug 22, 2026 by abhishekraok Contributor Loading…
2
Add OMEGA-500 task
#308 opened Aug 22, 2026 by abhishekraok Contributor Loading…
9
Fix ladder dataset resolution and vLLM runtime installs
#303 opened Aug 19, 2026 by AkshitaB Contributor Loading…
2
Add single-turn IFEval task
#297 opened Aug 17, 2026 by abhishekraok Contributor Loading…
4 of 5 tasks
1
Adding CTC Suite (upcoming release) to olmo eval
#295 opened Aug 17, 2026 by PrasannS • Draft
Fix Hugging Face stop-sequence decoding
#294 opened Aug 16, 2026 by sylvesterkaczmarek Loading…
1
Add FIRMCore evaluation task implementation
#292 opened Aug 15, 2026 by pardiszahraei Loading…
6 tasks done
feat: support prompt truncation across inference providers
#291 opened Aug 14, 2026 by sylvesterkaczmarek Loading…
6 of 13 tasks
1
feat: add weighted suite aggregation
#290 opened Aug 14, 2026 by sylvesterkaczmarek Loading…
6 of 13 tasks
1
Prasann/ctc suite
#289 opened Aug 14, 2026 by PrasannS Loading…
Add HELMET task suite: all seven categories, recall extended to 2m tokens
#281 opened Aug 14, 2026 by jopetty Contributor Loading…
2
Add multimodal evaluation enhancement New feature or request
#278 opened Aug 11, 2026 by jason718 • Draft
11 tasks done
@jason718 @uwGZQ
2
Add ruler-plus extending RULER evals to 2m tokens
#276 opened Aug 10, 2026 by jopetty Contributor Loading…
7
[Draft] Add StrongReject
#275 opened Aug 6, 2026 by mgmorgan23 Contributor Loading…
1 of 13 tasks
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.