You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**SciEval** is an open-source evaluation framework and leaderboard for measuring the *scientific intelligence* of large language and vision–language models.
4
-
It targets the full research workflow which is from scientific image understanding to hypothesis generation and provides a reproducible toolkit that unifies data loading, prompt construction, inference and evaluation.
3
+
<palign="center"><strong>
4
+
A unified evaluation toolkit and leaderboard for rigorously assessing the scientific intelligence of large language and vision–language models across the full research workflow.
**SciEval** is an open‑source evaluation framework and leaderboard aimed at measuring the **scientific intelligence** of large language and vision–language models.
34
+
Although modern frontier models often achieve *~90* on general‑purpose benchmarks, their performance drops sharply on rigorous, domain‑specific scientific tasks—revealing a persistent **general‑versus‑scientific gap** that motivates the need for SciEval.
35
+
Its design is shaped by following core ideas:
36
+
37
+
-**Beyond general‑purpose benchmarks ▸** Traditional evaluations focus on surface‑level correctness or broad‑domain reasoning, hiding models’ weaknesses in realistic scientific problem solving. SciEval makes this **general‑versus‑scientific gap** explicit and supplies the evaluation infrastructure needed to guide the integration of broad instruction‑tuned abilities with specialised skills in coding, symbolic reasoning and diagram understanding.
38
+
-**End‑to‑end workflow coverage ▸** SciEval spans the full research pipeline—such as **image interpretation, symbolic reasoning, executable code generation, and hypothesis generation**—instead of isolated subtasks.
39
+
-**Capability‑oriented & reproducible ▸** A unified toolkit for **dataset construction, prompt engineering, inference, and expert‑aligned scoring** ensures transparent and repeatable comparisons.
40
+
-**Grounded in real scenarios ▸** Benchmarks use domain‑specific data and tasks so performance reflects **actual scientific practice**, not synthetic proxies.
Modern frontier language models routinely score near *90* on general‑purpose benchmarks, yet even the strongest model (e.g., **Gemini 3 Pro**) drops below *60* when challenged by rigorous, domain‑specific scientific tasks. SciEval makes this **general‑versus‑scientific gap** explicit and supplies the evaluation infrastructure needed to guide the integration of broad instruction‑tuned abilities with specialised skills in coding, symbolic reasoning and diagram understanding.
46
+
47
+
## <imgsrc="assets/icon/progress.png"alt="progress"height="28"style="vertical-align:middle;" /> Progress in Scientific Intelligence
48
+
49
+
*Realtime updates — scores are synchronized with the [Intern‑Discovery‑Eval](https://opencompass.org.cn/Intern-Discovery-Eval/rank) leaderboard.*
|**Multimodal & Executable Scoring**| Supports text, code, and image inputs; integrates code tasks and LLM-judge fallback for open-ended answers. |
22
-
|**Reproducible & Extensible**| Clear dataset and model registries, minimised hard-coding and modular evaluators make new tasks or checkpoints easy to plug in. |
55
+
-**General benchmarks overestimate scientific competence.** Even the strongest frontier models (e.g., **Gemini 3 Pro**) score below **60** on **Scientific Text Capability** , despite scoring near *90* on widely used general‑purpose benchmarks.
56
+
-**Multimodal capability is breaking the 60‑point barrier.****Gemini 3 Pro** leads **Scientific Multimodal Capability** with **62.88**, reflecting strong performance in multimodal perception and reasoning.
57
+
-**Open‑source systems are rapidly closing the gap.***Qwen3‑VL‑235B‑A22B* and *Qwen3‑Max* now match or surpass several proprietary models in symbolic reasoning and code generation, signalling healthy community progress.
58
+
-**Symbolic reasoning and code generation remain bottlenecks.** No model exceeds **50** in equation‑level manipulation or **30** in end‑to‑end executable code tasks, indicating that scientific workflows requiring programmatic pipelines still fail frequently.
|**Multimodal & Executable Scoring**| Supports text, code, and image inputs; integrates code tasks and LLM-judge fallback for open-ended answers. |
68
+
|**Reproducible & Extensible**| Clear dataset and model registries, minimised hard-coding and modular evaluators make new tasks or checkpoints easy to plug in. |
31
69
32
-
• Coverage: five scientific capability dimensions × six major disciplines in the initial benchmark suite.
33
70
34
-
***Community Submissions Open**
35
-
Submit your benchmarks via pull request to appear on the official leaderboard.
***[2025‑12‑12] · 📰 Evaluation Published on OpenCompass**
73
+
• SciEval’s benchmark results are now live on the [OpenCompass](https://opencompass.org.cn/Intern-Discovery-Eval) platform, providing broader community visibility and comparison.
36
74
37
-
## Codebase Updates
38
-
***Execution‑based Scoring**
39
-
Code‑generation tasks (SciCode, AstroVisBench) are now graded via sandboxed unit tests.
75
+
***[2025‑12‑05] · 🚀 SciEval v1 Launch**
76
+
• Initial public release of a science‑focused evaluation toolkit and leaderboard devoted to realistic research workflows.
40
77
41
-
## Quick Start
78
+
• Coverage: seven scientific capability dimensions × six major disciplines in the initial benchmark suite.
79
+
***[2025‑12‑05] · 🌟 Community Submissions Open**
80
+
• Submit your benchmarks via pull request to appear on the official leaderboard.
42
81
43
-
Get from clone to first scores in minutes—see our local
Get from clone to first scores in minutes—see our local [QuickStart](docs/en/Quickstart.md) / [快速开始](docs/zh-CN/Quickstart.md) guides, or consult the [VLMEvalKit tutorial](https://vlmevalkit.readthedocs.io/en/latest/Quickstart.html) for additional reference.
SciEval ToolKit is built on top of the excellent **[VLMEvalKit](https://github.com/open-compass/VLMEvalKit)** framework and we thank the OpenCompass team not only for open‑sourcing their engine, but also for publishing thorough deployment and development guides ([Quick Start](https://vlmevalkit.readthedocs.io/en/latest/Quickstart.html), [Development Notes](https://vlmevalkit.readthedocs.io/en/latest/Development.html)) that streamlined our integration.
92
143
93
144
We also acknowledge the core SciEval contributors for their efforts on dataset curation, evaluation design, and engine implementation: Jun Yao, Han Deng, Yizhou Wang, Jiabei Xiao, Jiaqi Liu, Encheng Su, Yujie Liu, Weida Wang, Junchi Yao, Haoran Sun, Runmin Ma, Bo Zhang, Dongzhan Zhou, Shufei Zhang, Peng Ye, Xiaosong Wang, and Shixiang Tang, as well as all community testers who provided early feedback.
145
+
146
+
SciEvalKit contributors can join the author list of the report based on their contribution to the repository. Specifically, it requires 3 major contributions (implement a new benchmark, foundation model, or contribute a major feature). We will update the report quarterly and an additional section that details each developer’s contribution will be appended in the next update.
0 commit comments