Formatting for Code Extractor

[TheMikeMerrill]

Hi there, quick question about curator.CodeExtractor.

What's the proper way to handle multiple test cases for the same prompt? I was looking at this example and it looks like code_input should return a string with each input on a separate line, and code_output should do something similar.

Also, how should my tested code accept these inputs? Via stdin? Thanks!

Here's my executor for reference:

class CodeGolfTestExecutor(curator.CodeExecutor):
    """Test Executor."""
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.parsed_code = None

    def code(self, row):
        """Extract code string from a dataset row."""
        
        code = row["deepseek_solution_extracted"]
        return code

    def code_input(self, row):
        """Extract single input from a dataset row."""
        inputs = row["test_inputs"]
        inputs = "\n".join([str(i) for i in inputs])
        return inputs
  

    def code_output(self, row, execution_output):
        """Parse execution results."""
        output = row["test_outputs"]
        output = "\n".join([str(o) for o in output])
        row["correct"] = output == execution_output.stdout
        return row

[shreyaspimpalgaonkar]

Hey Mike, thanks for your question!

We wanted to kept the CodeExtractor abstractions fairly general, so that they aren't too specific to competitive programming code generation tasks. If you want to run it on a list of test cases, you can do one of these two things:

1. change the code generation instruction to accept input for a list of test cases (so that it generates a for loop in the code itself)
2. if you've already generated code using deepseek, you can pass it through curator with a cheaper model and ask it to modify the code and add the for loop.