LiveCodeBench V5 Code and Eval Score

[Andrewzh112]

I just tested on LCB V5 with improvised code and got low scores compared to the reported V4 scores, can you give an example script on evaling on LCB V5? Thanks

[Andrewzh112]

to add more context, I added the following LLMStyle for acecoder:

  if LanguageModelStyle == LMStyle.AceCoder:
      prompt = f"""Please provide a self-contained Python script that solves the following problem in a markdown code block:

{get_generic_question_template_answer(question)}

"""
      TEMPLATE_INPUT_PROMPT = """Below is a Python script with a self-contained function that solves the problem and passes corresponding tests:
```python"""
      chat = [
          {"role": "user", "content": prompt},
          {"role": "assistant", "content": TEMPLATE_INPUT_PROMPT},
      ]
      from transformers import AutoTokenizer
      tokenizer = AutoTokenizer.from_pretrained(
          "TIGER-Lab/AceCoder-Qwen2.5-7B-Ins-Rule", padding_side="left", use_fast=False
      )
      return tokenizer.apply_chat_template(
          chat,
          tokenize=False,
          add_generation_prompt=False,
          truncation=False,
          padding=False,
      )

[wyettzeng]

Hi Andrew, instead of adding a new LMStyle, you should just use LMStyle.CodeQwenInstruct. We have not tested on V5 as that isn't available when we ran our experiment. However, it does seem that from their official leaderboard, v5 is harder than v4 which explains the drop in scores. To test on V5, after you have added an entry in lcb_runner/lm_styles.py for AceCoder with LMStyle.CodeQwenInstruct, just use their native code:

python -m lcb_runner.runner.main --model $model --scenario codegeneration --evaluate --release_version release_v5

Here are screenshots from their leaderboard (you can see that O3-Mini-2025-01-31 (High) also dropped from v4 to v5):
V4:

V5: