Allow `.statement` files for test cases in `Secret`

[hairez]

We should allow .statement files in Secret. This could be useful for problems with public feedback for Secret test cases, where it shows .statement files instead of .ans or .out.

Additionally, it is currently not allowed to have both an .out file and an .ans.statement file in the Sample.
It says Note that it is an error to provide both a `.out` and a `.ans.statement` file..

The reason it is like this, is because .ans.statement will be the one that is shown, and it would be pointless to validate .out files in that case.
If this is a restriction in Sample, it should also be a restriction in Secret.

That said, I’m leaning towards removing the restriction altogether and allowing both, so the problem setter has more flexibility. In my opinion, it would not hurt to validate some extra .out that is not displayed.

[RagnarGrootKoerkamp]

I'm not against this, but I think it would be good to have a concrete example of where data/secret/testcase.{in,ans}.statement truely helps over simply showing the plain .{in,ans}.

I think the restriction of not doing both .ans.statement and .out at the same time is good. This avoids a bunch of confusion.

[hairez]

A concrete example is this:

https://open.kattis.com/problems/asimplegame

The output is too long to be shown, so one would like to use ... to make it readable as an .ans.statement. However, one should not be able to output validate this .ans.statement.

[RagnarGrootKoerkamp]

Ah ok I got confused by the >1 points in the OP, but yes, sounds reasonable to have this for feedback of secret cases as well.

[mzuenni]

Note that it is an error to provide both a .out and a .ans.statement file.

I think we should keep this restriction. If we have both files, it is not clear what should be displayed, and this will probably confuse users. It is just much cleaner and less error-prone to forbid this.

[niemela]

That said, I’m leaning towards removing the restriction altogether and allowing both, so the problem setter has more flexibility. In my opinion, it would not hurt to validate some extra .out that is not displayed.

I'm not against removing this restriction, but is it ever useful to have both .out and .ans.statement.

The argument for disallowing it was that it "doesn't do anything", and that it's easier to understand the spec when things that are completely useless are disallowed. But, you can also make an argument that having fewer rules and exceptions makes it easier to understand. (I think I'm buying the second argument...)

I think we should keep this restriction. If we have both files, it is not clear what should be displayed, and this will probably confuse users. It is just much cleaner and less error-prone to forbid this.

I think it is clear, but if it isn't we should just make it clear. The rule is very simple, you should always display .ans.statement (int the statement) if it exists, otherwise .out if it exists, otherwise .ans.

I could see a good argument for maybe naming it .out.statement though. Would that make it clearer @mzuenni?

[mzuenni]

I could see a good argument for maybe naming it .out.statement though. Would that make it clearer?

not sure. i think .ans.statement is better because typically the .ans is displayed and it feels wrong to have .out.statement without .out (which would be the case most of the time)