Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
162 Open 1,700 Closed
162 Open 1,700 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Multiple Bangla Benchmark datasets added
#3454 opened Dec 9, 2025 by Ismail-Hossain-1 Loading…
Add Uncheatable Eval
#3442 opened Dec 2, 2025 by ziqing-huang Loading…
1
Refactor TaskManager
#3432 opened Nov 26, 2025 by baberabb Loading…
Adding pisa task
#3412 opened Nov 17, 2025 by HallerPatrick Loading…
3
Fix gsm8k_platinum description
#3411 opened Nov 17, 2025 by fxmarty-amd Loading…
1
2
Fix wrong gpqa_diamond_generative_n_shot answer template
#3407 opened Nov 15, 2025 by fxmarty-amd Loading…
1
Fix MBPP task configuration and code extraction logic
#3387 opened Nov 6, 2025 by tanios13 Loading…
4
feat: refine Chain-of-Thought removal logic
#3386 opened Nov 6, 2025 by Co-Cl2 Loading…
4
[feat] Add Countdown Task
#3384 opened Nov 4, 2025 by StephenXie Loading…
7
Math 500
#3381 opened Nov 1, 2025 by seldereyy Loading…
4
Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py
#3372 opened Oct 28, 2025 by vnayakde Loading…
2
Add support for configurable chrF metric parameters in task YAML, fix…
#3363 opened Oct 23, 2025 by augustlakia Loading…
4
Add gsm_symbolic and gsm_symbolic_cot tasks
#3354 opened Oct 19, 2025 by MengAiDev Loading…
1
[AIME24 | AIME25] Enable Multiple Generation Repeats with Pass@k and Majority@k Metrics
#3351 opened Oct 17, 2025 by ihebchaa Loading…
11
Added ULQA benchmark
#3340 opened Oct 13, 2025 by keramjan Loading…
1
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
2
Support empty response for Completions and ChatCompletions API
#3309 opened Sep 22, 2025 by tboerstad Loading…
7
Adding New Task SLR-Bench : Scalable Logical Reasoning Benchmark
#3305 opened Sep 20, 2025 by Ahmad21Omar Loading…
2
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
Gemini evaluation support
#3300 opened Sep 15, 2025 by IsraelAbebe Loading…
2
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
Add long-context evaluation benchmarks (LongBench v2, Babilong, InfiniteBench, Phonebook)
#3256 opened Aug 21, 2025 by Mariani-code Loading…
1
11
fix gsm8k normalization
#3254 opened Aug 20, 2025 by huaanrui Loading…
1
Main
#3250 opened Aug 20, 2025 by seongtaehong Loading…
3
Adding 3LM to lm eval harness
#3241 opened Aug 14, 2025 by GeorgeSherif Loading…
1
ProTip! Type g i on any issue or pull request to go back to the issue listing page.