When is a testcase rejected?

[mzuenni]

The spec currently specifies:

Each tc.in under invalid_input must be rejected by at least one input validator.

There are two problems with this:

1. I think for input validators we do not specify what rejection means. It should be defined as "exits with exit code 43"
2. Is it ok for validators to crash if at least one rejected? If yes we may want to mention this.

I would argue in favor of 2.

For example, consider a problem where the input is supposed to be a positive integer x with the extra property that 42 / x is also an integer.
Now one checker can check that its a positive integer and the other can simply check that 42 & x == 0.
However, the second checker will crash if x is zero. IMO this is fine since the other checker already ensures that x is not zero.

[hairez]

I don’t fully understand the concern here.

Is it really a problem if the input validator “crashes” on invalid input? For example, many input validators use assert(), which technically causes a crash on failure. But I agree that this should be mentioned.

However, the new spec and legacy seem to cover this case:

The input validator must exit with code 42 on successful validation. Any other exit code means that the input file could not be confirmed as valid.

[mzuenni]

Is it really a problem if the input validator “crashes” on invalid input? For example, many input validators use assert(), which technically causes a crash on failure.

Yes, and I think this is bad. This makes it impossible to distinguish between the validator telling us that the file is invalid and it having a bug. For normal test cases this is not much of an issue because the validator should neither reject nor crash on those.
However, we introduced invalid_(in|out)puts to verify the behavior of the checkers. And IMO it would be nice to ensure here that the validators do not crash but properly reject a test case (i.e. exits with 43).

[niemela]

Yes, and I think this is bad. This makes it impossible to distinguish between the validator telling us that the file is invalid and it having a bug. For normal test cases this is not much of an issue because the validator should neither reject nor crash on those.
However, we introduced invalid_outputs to verify the behavior of the output checker. And IMO it would be nice to ensure here that the output validator does not crash but properly reject a test case (i.e. exits with 43).

Yes, exactly!

[evouga]

So are we talking about input validators or output validators?

Imo:

• at least one input validator should cleanly exit with a non-42 error code on invalid input. It is ok for other input validators to crash;
• the output validator should cleanly exit with error code 43 on invalid output.

The rationale for the former is that it is often best to validate inputs using multiple validators that test for different things: a .ctd to check syntax, and then a .py for further checking assuming valid syntax, e.g. as @mzuenni explained.

What one really cares about when constructing the set of input validators is that they collectively detect malformed input.

The output validator must be bulletproof for obvious reasons.

[mzuenni]

So are we talking about input validators or output validators?

input validators

at least one input validator should cleanly exit with a non-42 error code on invalid input. It is ok for other input validators to crash;

Yes, but i would define at least one exits cleanly as: at least one exits with 43, else there is no chance to spot ununtended crashes.

[evouga]

Exit code 43 is an output-validator thing. There is nothing special about 43 in input validators (and indeed in practice a variety of return values are used to signal which part of validation failed).

[mzuenni]

Exit code 43 is an output-validator thing.

Yes, and I propose to also make it an input validator thing

There is nothing special about 43 in input validators

Yes, so far (except that it is clearly distinct from common exit codes produces by crashes)

The invalid_inputs directory becomes a lot more useful if we can properly distinguish between stuff crashing and stuff getting intentionally rejected.

Besides that, we (BAPCtools) already used this convention in the past, and it helped us find bugs where input validators crashed unintentionally. Additionally, this also makes debugging rejected test cases easier (if the exit code is 43 the test case is probably wrong if the exit code is something else the checker is probably buggy).

[evouga]

So:

• requiring input validators to return 43 when rejecting invalid input: that sounds fine to me. For well-formed legacy packages this makes no difference (since all input validators return 42 on all inputs, presumably). Do CheckTestData and Viva already conform to this convention? (If not, we will need buy-in from them, no?)
• defining what it means for invalid input to be rejected: at least one input validator return 43, and the rest can do anything (including returning 42 or crashing)?

[evouga]

It looks like CTD and VIVA already don't follow the conventions (they both return 0 to indicate successful validation, e.g.). So I suppose they are handled separately.