Any point in this benchmark anymore?

[EdanToledo]

Hey, I had a quick question regarding the existence of this benchmark? If we are just adding methods to the leaderboard where people climb on the test set, what is the point of any of this? I am just wondering if the maintainers made a mistake with this? or if they care? I think if they care, it would be wise to at least add a disclaimer that some of the results on the leaderboard are from agents that hillclimb the test set.

[joe-needham]

Thanks for raising this. In hindsight I do agree it's reasonable to add a disclaimer when authors admit their agent makes use of test-label leakage, even if we can't apply this to past submissions (or guarantee future submissions disclose this). Hopefully this works as a short-term solution: #125.

We hope to have more bandwidth soon to fix some of these exploitability issues in the benchmark (https://github.com/openai/mle-bench?tab=readme-ov-file#known-issues), then archive and replace the current leaderboard.

[timeleft--]

Hi! Following this thread with interest. I may be misreading, but it seems like two different concerns are being conflated here. The issue raises a question about test-set hillclimbing -- whether the evaluation protocol incentivizes iterating on test feedback. PR #125 adds a "Known Leakage" column, which is about data leakage (label contamination, exploit-prone tasks, etc.). These are distinct issues. Hillclimbing is a property of the protocol; leakage is a property of a certain set of tasks that has known issues.

How does the Known Leakage column capture the concern raised here? @joe-needham

Thanks!

[AtrixTang]

In recent submissions, specifically in PR #119 and PR #118, the authors explicitly acknowledged that their methods involved iterating on the test set to improve their final scores. Despite this clear admission, these submissions were still approved and added to the official leaderboard by the maintainers.
This raises a few critical questions:

• Is this fair to other participants who strictly adhere to standard machine learning practices by keeping the test set completely unseen?
• Does this align with the original intention of mle-bench, which was presumably designed to rigorously evaluate genuine machine learning engineering capabilities rather than the ability to overfit a specific set of test cases?

If the leaderboard inadvertently encourages or rewards test set iteration, it severely risks undermining the credibility, fairness, and overall utility of the benchmark. I hope the maintainers can clarify the official stance on this matter. @joe-needham @thesofakillers

[joe-needham]

@timeleft-- thanks for clarifying here, I think you’re right I was conflating the two and incorrectly assumed issues in the tasks themselves (exploiting test-set leakage, e.g. known issues) was the common theme of discussions in PR #119 and PR #118.

I agree using the test-set in this way is not standard practice in ML. Unless there’s a strong reason for MLE-bench being an exception here that I'm missing, I’d be in favour of the leaderboard not including these entries. I’m also not sure if a separate leaderboard for these entries is a good idea, since again it’s not usual ML practice to use the test-set in this way. I understand it could be motivated by Kaggle’s system, but I don’t think that holds here because the final performance is always measured on a held-out set as @AtrixTang mentioned.

Sorry for the oversight. Going forward, we’ll make this explicit by requiring each submission to attest in their PR that the test-set isn’t used in this way, and further encourage open-sourcing code.

[moustafa-a]

Hi Moustafa here from the disarray team. I don't think it is accurate to characterize our approach as hill-climbing or oracle-guided optimization. A small number of medal / no-medal signals yields at most a few bits total: no gradient, no magnitude, no notion of closeness to the threshold, and no error localization. On a combinatorial ML engineering problem with very high degrees of freedom (model choice, features, hyperparameters, pipelines, ensembling), that signal is a far-cry from oracle-guided optimization.

This is a mechanism that we introduced for resource management. In our trials, the agents achieved a medal within ~7.4 hours on average, on 1× A100 GPU. Early termination allowed us to greatly improve resource efficiency for conducting experiments. One could argue that the usage of H100s and H200s for experimentation has a much bigger impact on the results, and resource type is not explicitly called out in the current leaderboard.

Ultimately, it is a question to the benchmark creators on what purpose it serves. Our participation was intended to encourage exchange of ideas on the underlying approaches, so that we as a community could advance towards highly capable MLE agents that can generate real-world impact. We have been transparent about our approach and are actively working on the technical report, which we’ll share as soon as possible.

[Brawn9913]

The justification still fundamentally conflicts with standard ML evaluation principles, because the evaluation protocol differs so fundamentally from standard zero-shot evaluation, these results are simply not apples-to-apples with strict pass@1 submissions.

1. A 1-bit Oracle is Still an Oracle
   Arguing that a binary signal provides "no gradient" and "only a few bits" misses the core issue: selection bias. If an agent generates multiple pipelines and uses the private test set to filter out the ones that fail to reach Bronze, it is explicitly selecting on the test set. In a high-degree-of-freedom space, using the final exam to validate your choices is the definition of oracle-guided optimization.

2. Resource Management Cannot Justify Data Leakage
   If the goal is early termination to save compute, the standard ML practice is to split a Validation Set from the training data. Using the Private Test Set as a stopping criterion is a methodological flaw, not a resource optimization technique.

3. Aligning with Real-World ML Practices
   While I completely agree with your vision of advancing agents for "real-world impact," true real-world ML engineering relies heavily on the strict separation of development and deployment data. Adhering strictly to held-out evaluation on MLE-bench is exactly what ensures these agents will be reliable when deployed in the real world.