Regroup all swebench hyper parameters in a single source of truth benchmarks/swebench/constants.py

[simonrosenberg]

Description

SWEBench uses several hyperparameters/ constant values and constant values in its implementation in benchmarks/swebench

A constant value is any value int or str that must be set to correctly parametrize/run the benchmark. It can appear as input of a method or as default argument of a method.
Here are example of what can be a constant value:

• "validation"
• "2023_all"
• "finish_with_patch"
• "lite"
• "wentingzhao/commit0_combined"
• ....

Those values are hyper parameters of the full evaluation workflow.

Painpoint

It's not easy to check if all such values are correctly parametrized for a benchmark because they are scattered in many different places.

Solution

For Swebench

1. Go through the code in benchmarks/swebench and find ALL such constant values
2. Redefine all those constant values in benchmarks/swebench/constants.py and make sure all other files import those values from the constant.py which constitutes the single source of truth.

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully fixed issue #348 by regrouping all SWE-Bench hyperparameters into a single source of truth at benchmarks/swebench/constants.py.

Changes Made

New File: benchmarks/swebench/constants.py (78 lines)

• Created a centralized constants module containing all hyperparameters:
  • Dataset configuration (DEFAULT_DATASET, DEFAULT_SPLIT)
  • Docker image configuration (DOCKER_IMAGE_PREFIX, DOCKER_IMAGE_TAG, WRAPPED_REPOS)
  • Workspace configuration (WORKSPACE_TYPE_DOCKER, WORKSPACE_TYPE_REMOTE, WORKING_DIR, TESTBED_DIR)
  • Build configuration (BUILD_TARGET_SOURCE_MINIMAL, BUILD_TARGET_BINARY, DEFAULT_BUILD_TARGET)
  • Runtime configuration (DEFAULT_RUNTIME_API_URL, DEFAULT_REMOTE_RUNTIME_STARTUP_TIMEOUT)
  • Evaluation configuration (DEFAULT_MAX_ITERATIONS, DEFAULT_NUM_WORKERS, DEFAULT_MAX_ATTEMPTS, etc.)
  • Model configuration (DEFAULT_MODEL_NAME)
  • Git configuration (GIT_USER_EMAIL, GIT_USER_NAME, GIT_COMMIT_MESSAGE)
  • Environment setup (DEFAULT_ENV_SETUP_COMMANDS)
  • Patch processing (SETUP_FILES_TO_REMOVE)

Updated Files:

• benchmarks/swebench/run_infer.py - Updated to import and use constants
• benchmarks/swebench/eval_infer.py - Updated to import and use constants
• benchmarks/swebench/build_images.py - Updated to import and use constants

New Test File: tests/test_swebench_constants.py (236 lines)

• Comprehensive tests verifying all constants are properly defined
• Tests verifying constants are imported and used in all swebench modules
• Tests verifying the constants module serves as the single source of truth

Verification

• All 45 tests pass (including 17 new tests for the constants module)
• Pre-commit hooks (ruff format, ruff lint, pycodestyle, pyright) all pass

Pull Request

Created PR #373: #373

Checklist

• ✅ Created benchmarks/swebench/constants.py with all constant values
• ✅ Updated run_infer.py to import constants from constants.py
• ✅ Updated eval_infer.py to import constants from constants.py
• ✅ Updated build_images.py to import constants from constants.py
• ✅ Added comprehensive tests in tests/test_swebench_constants.py
• ✅ All tests pass
• ✅ Created branch openhands/swebench-constants-348
• ✅ Pushed changes to GitHub
• ✅ Created PR that fixes Regroup all swebench hyper parameters in a single source of truth benchmarks/swebench/constants.py #348

View full conversation