Test tools report "All tests passed!" despite failing test cases.

[LeeJ-art]

Dear SQIsign team,

My name is LeeJ, and I am working on implementing and optimizing the SQIsign reference code on an Arm architecture. While checking the code's correctness, I've come across an issue where the test suite reports "All tests passed!" even when some test cases fail.

Bug

In src/signature/ref/lvlx/test/test_signature.c, the test script reports "All tests passed!" even when certain test cases fail.

Solution

We should change the value of res to 0, which prevents the main function from printing "All tests passed!" if any cases fail.

int test_sqisign(int repeat){
    int res = 1;

    public_key_t pk;
    secret_key_t sk;
    signature_t sig;
    const unsigned char msg[32] = { 0 };

    public_key_init(&pk);
    secret_key_init(&sk);

    printf("\n\nTesting signatures\n");
    for (int i = 0; i < repeat; ++i) {
        printf("#%d \n", i);

        protocols_keygen(&pk, &sk);
        protocols_sign(&sig, &pk, &sk, msg, 32);
        int check = protocols_verify(&sig, &pk, msg, 32);
        if (!check) {
            // Here
            res = 0;
            printf("verif failed ! \n");
        }
    }

    public_key_finalize(&pk);
    secret_key_finalize(&sk);

    return res;
}

Additional question

I've been trying to quickly check the correctness of my Level 1 code. I've tried running make test SQIsign_lvl1_SELFTEST, but this seems to take an amount of time. I also attempted to run the test_signature.c file directly, but it seems to not guarantee full correctness.

Could you please provide some guidance on a faster or more direct method to check the correctness of the Level 1 signature implementation?

[defeo]

Thanks for the report. Regarding your additional question, I'm not aware that make test XXX does anything different from make test (indeed, you miscapitalized the name of the test). If you want to run only that single test you can type ctest -R sqisign_lvl1_SELFTEST, and that's almost instantaneous for me (0.19 seconds).

[LeeJ-art]

Thanks for your kind reply. My apologies for the capitalization error; I was following the name as it appears in the README. Thank you for pointing out that ctest -R sqisign_lvl1_SELFTEST is the correct command to run a single test.