Evaluate openevolve on ALE Bench

[codelion]

https://sakana.ai/ale-bench/

[Yuki-Imajuku]

Hi! Thank you for your interest in our ALE-Bench!
Although I'm not familiar with OpenEvolve, but I can run one ALE-Bench problem (ahc046) with the files below. I can't work on this integration, but I'm commenting here to support your contribution :)
This example is not sophisticated, so please don't persist with this temporary solution!!!

Environment

C6i EC2 Instance (Ubuntu)
I used uv to create the OpenEvolve environment and then I add ale-bench

git clone https://github.com/codelion/openevolve.git
cd openevolve
uv sync
uv add https://github.com/SakanaAI/ALE-Bench.git

Files

• examples/ale_bench/config.yml

# Configuration for function minimization example
max_iterations: 5
checkpoint_interval: 1
log_level: "INFO"

# LLM configuration
llm:
  primary_model: "o4-mini-2025-04-16"
  primary_model_weight: 0.8
  secondary_model: "gpt-4.1-mini-2025-04-14"
  secondary_model_weight: 0.2
  temperature: 1.0
  top_p: 1.0
  max_tokens: 32768
  timeout: 300

# Prompt configuration
prompt:
  system_message: |
    You are a world-class algorithm engineer, and you are very good at programming. Now, you are participating in a programming contest. You are asked to solve a heuristic problem, known as an NP-hard problem.

    [Problem statement]
    Execution time limit: 2.0 sec / Memory limit: 1024 MB

    Problem Statement
    ...(problem statement from ahc046 here)...
  num_top_programs: 1
  num_diverse_programs: 2
  use_template_stochasticity: false

# Database configuration
database:
  db_path: /path/to/openevolve/examples/ale_bench/database
  in_memory: false
  population_size: 25
  archive_size: 25
  num_islands: 5
  migration_interval: 2
  migration_rate: 0.2
  elite_selection_ratio: 0.1
  exploration_ratio: 0.3
  exploitation_ratio: 0.6
  feature_dimensions:
    - "score"
    - "complexity"
  feature_bins: 10

# Evaluator configuration
evaluator:
  timeout: 300
  cascade_evaluation: false
  parallel_evaluations: 1
  use_llm_feedback: false

# Evolution settings
diff_based_evolution: true
allow_full_rewrites: false

• examples/ale_bench/evaluator.py

import traceback
from pathlib import Path

import ale_bench
from ale_bench.result import CaseResult, JudgeResult, Result


root_dir = Path(__file__).resolve().parent
session_file = root_dir / "session.json"


def result_feedback(result: Result) -> CaseResult:
    if result.overall_judge_result == JudgeResult.ACCEPTED:
        return result.case_results[0]
    else:
        selected_case_idx = 0
        for idx, case_result in enumerate(result.case_results):
            if case_result.judge_result == result.overall_judge_result:
                selected_case_idx = idx
                break
        return result.case_results[selected_case_idx]


def evaluate(program_path):
    try:
        session = None
        if not session_file.exists():
            session = ale_bench.start(
                problem_id="ahc046",
                num_workers=13,
            )
        else:
            session = ale_bench.restart(session_saved_file=session_file, num_workers=13)
        if not session:
            raise RuntimeError("Failed to start or restart the session.")

        code = Path(program_path).read_text().replace("# EVOLVE-BLOCK-START", "").replace("# EVOLVE-BLOCK-END", "").strip()
        public_result = session.public_eval(code=code, code_language="cpp23")
        extracted_case = result_feedback(public_result)
        session.save(session_file)

        return {
            "judge_result": public_result.overall_judge_result.value,
            "overall_score": public_result.overall_absolute_score,
            "max_execution_time_sec": max([case_result.execution_time for case_result in public_result.case_results]),
            "max_memory_usage_mib": max([case_result.memory_usage for case_result in public_result.case_results]) // 1024 // 1024,
            "standard_error": extracted_case.error_str,
            "message": extracted_case.message,
        }
    except Exception as e:
        print(f"Evaluation failed completely: {str(e)}")
        print(traceback.format_exc())
        return {
            "overall_score": 0.0,
            "error": str(e),
        }

• initial_program.cpp

# EVOLVE-BLOCK-START
#include <bits/stdc++.h>
#include <atcoder/all>
using namespace std;
using namespace atcoder;

int main() {
    return 0;
}
# EVOLVE-BLOCK-END

Run

OPENAI_API_KEY="sk-..." .venv/bin/python openevolve-run.py examples/ale_bench/initial_program.cpp examples/ale_bench/evaluator.py --config examples/ale_bench/config.yaml

Result

• examples/ale_bench/openevolve_output/best/best_program.cpp

I think this solution is a very simple greedy one.

# EVOLVE-BLOCK-START
#include <bits/stdc++.h>
#include <atcoder/all>
using namespace std;
using namespace atcoder;

int main() {
    ios::sync_with_stdio(false);
    cin.tie(nullptr);

    int N, M;
    cin >> N >> M;
    vector<pair<int,int>> pts(M);
    for (int i = 0; i < M; ++i) {
        cin >> pts[i].first >> pts[i].second;
    }

    ...(omit)...

    // Output all moves
    for (auto &p : ans) {
        cout << p.first << ' ' << p.second << "\n";
    }
    return 0;
}
# EVOLVE-BLOCK-END

• examples/ale_bench/openevolve_output/best/best_program_info.json

{
  "id": "c5d424dc-c68f-41b2-a866-805ea3ec7140",
  "generation": 2,
  "iteration": 2,
  "timestamp": 1750158033.155996,
  "parent_id": "6de7381d-38c3-4a03-a7d8-cc0d20a5abf6",
  "metrics": {
    "judge_result": "ACCEPTED",
    "overall_score": 67982,
    "max_execution_time_sec": 0.0,
    "max_memory_usage_mib": 3,
    "standard_error": "",
    "message": ""
  },
  "language": "cpp",
  "saved_at": 1750158208.9139886
}

[codelion]

Thanks for providing this, I think this is a great start for implementing the full harness for ALE-Bench,

[BradKML]

What other benchmarks could be included for OpenEvolve? MLE-Bench? RE-Bench? Anything SWE related?

[rujialiu]

What other benchmarks could be included for OpenEvolve? MLE-Bench? RE-Bench? Anything SWE related?

As a former competitive programmer, personally I'm very much interested in the (harder) problems in LiveCodeBenchPro https://livecodebenchpro.com/ It should be not hard to integrate.

[ofirpress]

Thanks @BradKML for posting AlgoTune to this thread :)

[BradKML]

Anything that moves the needle (for the GPU-poor), @ofirpress