how to interpret evaluation result

[yxuansu]

hey! thanks for releasing the code. i tried to run a demo test and get the following results in accuracy.json

{
    "overall": {
        "accuracy": 0.6212443095599393,
        "correct": 2047,
        "total": 3295
    },
    "by_dimension": {
        "conditional": {
            "accuracy": 0.4105263157894737,
            "correct": 78,
            "total": 190
        },
        "unconditional": {
            "accuracy": 0.6466489714664897,
            "correct": 1949,
            "total": 3014
        },
        "example_driven": {
            "accuracy": 0.21978021978021978,
            "correct": 20,
            "total": 91
        }
    }
}

would like to check how to interpret the numbers corresponding to the metrics proposed in the paper.

ty!

[kijlk]

hey! thanks for sharing the results.
To interpret the numbers:

• total refers to the total number of constraints evaluated.
• accuracy corresponds to the CSR (Constraint Success Rate) reported in the paper.
• conditional matches the condition category in the paper.
• unconditional corresponds to the vanilla category.
• example_driven corresponds to the example category.

Thanks for reporting this! I’ve updated the repository so that the result names now match those used in the paper, and I’ve also added reports for the other categories to make the outputs fully consistent with the paper.