Skip to content

Rename ScholarQABench2 to ScholarQA-CS2 #129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
amanpreet692 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Rename ScholarQABench2 to ScholarQA-CS2 #129

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6cf70e6
Rename ScholarQABench2 to ScholarQA-CS2
amanpreet692 Nov 12, 2025
51c8f68
Fix code block formatting in README.md
amanpreet692 Nov 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ See [agent-baselines](https://github.com/allenai/agent-baselines) for more examp

Use `inspect eval` directly to run a single task. E.g. for the ScholarQA task:

``bash
```bash
uv run inspect eval --solver generate --model openai/gpt-4.1 astabench/sqa_dev
```

Expand Down Expand Up @@ -332,7 +332,7 @@ AstaBench includes a comprehensive suite of evaluation tasks across four main ca
Tasks that evaluate agents' ability to search, analyze, and synthesize academic literature:

- **PaperFindingBench** (`astabench/paper_finder_{validation,test}`): Find relevant papers for given research queries
- **ScholarQABench2** (`astabench/sqa_{dev,test}`): Answer scientific questions using literature search
- **ScholarQA-CS2** (`astabench/sqa_{dev,test}`): Answer scientific questions using literature search
- **LitQA2-FT** (`astabench/litqa2_{validation,test}`): Answer questions about scientific literature with binary responses
- **LitQA2-FT-Search** (`astabench/paper_finder_litqa2_{validation,test}`): Find papers relevant to LitQA2 questions
- **ArxivDIGESTables-Clean** (`astabench/arxivdigestables_{validation,test}`): Create structured tables summarizing information from arXiv papers
Expand Down