[EVAL] Long Horizon Execution

[NathanHB]

Evaluation short description

• Long context
• used by Jan moodel https://huggingface.co/janhq/Jan-v2-VL-low

Evaluation metadata

Provide all available

• Paper url: https://arxiv.org/abs/2509.09677
• Github url: https://github.com/long-horizon-execution/measuring-execution/
• Dataset url: https://huggingface.co/datasets/arvindh75/Long-Horizon-Execution

[akshathmangudi]

hi @NathanHB! i was browsing through the repo's open issues and stumbled across this issue. can i get some more information on the issue?

upon reading the links you sent me, i assume that you want the paper to be implemented?

i could work on this after i get some information.

[NathanHB]

hey @akshathmangudi ! Yep exactly, there is a nice codebase but it would be extremly useful if you added it :)

The idea is to use inspect-ai to implement the task, no need to other backends for now !

You can look at mmlu-pro on how to implement a task with inspect-ai in lighteval

[akshathmangudi]

i see!

if it's fine, i can work on this and drop an implementation approach in a few hours and tag you so you can review it!

[NathanHB]

sounds good !

[akshathmangudi]

Ideation

So upon loading the dataset, I noticed that there are 50,000 entries per example, so for an initial effort, it might be better to truncate the dataset to the first 100 or 1000 examples and scale-up.

Next, we will start with the single-turn approach and implement multi-turn later in the picture, as these approaches help in calculating the cumulative responses.

We will then construct a dictionary with input and value columns from the dataset and format the prompt with the dictionary and key sequence. We will follow the repo's prompt structure and ask for cumulative sums in <answer> tags.

We will use inspect-ai for the evaluation framework, which involves loading the dataset, sending prompts to model, scoring responses and managing the evaluation loop.

@NathanHB

[shash42]

Author of that evaluation here, @viciousAegis and I are quite happy to help if there are any questions :)

[akshathmangudi]

@NathanHB revisiting this to be closed 🤗