feat: add SWE-bench dataset support #2387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

peteli25 wants to merge 1 commit into open-compass:main from peteli25:feature/swebench-dataset

peteli25 commented Jan 14, 2026

gh pr create --title "feat: add SWE-bench dataset support" --body "## Description

This PR adds support for SWE-bench dataset evaluation in OpenCompass.

Changes

Add `swebench.py` dataset implementation with SWEBenchDataset class
Add `swebench_gen.py` configuration for dataset loading
Add `eval_swebench.py` evaluation configuration
Add `vllm_swe_llama_7b.py` model configuration for SWE-Llama
Update `registry.py` to register new components
Update `datasets/init.py` to export SWEBenchDataset

Features

Support for loading SWE-bench dataset from HuggingFace
Custom evaluator for SWE-bench code generation tasks
Compatible with vLLM inference backend

Usage

```bash
python run.py configs/eval_swebench.py
```
"


          feat: add SWE-bench dataset support

5f1295e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet