Can input validators be run on nextpass.in files

[Matistjati]

I see no reason why not (in fact, I think we should do that). But an implementer may not consider it if we don't explicitly specify that it should happen.

[RagnarGrootKoerkamp]

hmm; then we have to spec exactly how that would look like. I imagine an input validator might also want access to all previous .in files in order to judge whether the current one is valid?

But yeah generally this sounds reasonable, and probably validating them standalone is already valuable (at least for syntax).

cc @mzuenni

[mzuenni]

I think access to previous .in files is not needed. If a submission can't verify that an input is valid your problem is already broken?.
(Sure this allows funny stuff like redoing a pass by just given an old input again, but this is undetectable by submissions and therefore fine).

But I am not sure if we need that extra checker. The output validator must already check the user output before producing the nextpass.in from it...

[RagnarGrootKoerkamp]

Ah right good point on submission input being stateless, and thus the input validator also being stateless. Makes sense.

So then the question is mostly if multipass problems should have input validators at all? Since all input is generated.
I don't think it hurts to support this? Since the interactor may be long and complex, while the input validator could be short and much easier to manually verify.

[mzuenni]

One additional remark: For many problems, it will be impossible to write a proper input validator. What I mean by that is that we can only check the format, but that is not sufficient to say that the input is valid... From the submissions' perspective, the input is only valid if it was derived from the submissions output, but checking that is impossible.

For example, if we allow the submission to write one value from {0, 1, 2} but the submission knows that it only ever writes 0 or 1 the input 2 is invalid for this submission.

[niemela]

So then the question is mostly if multipass problems should have input validators at all? Since all input is generated.

The initial input is given, so should be validated, right?

I don't think it hurts to support this? Since the interactor may be long and complex, while the input validator could be short and much easier to manually verify.

Agreed. So multipass problems should have input validators, right?

[Zazmuz]

I think that each nextpass.in should be validated the same way as the initial input, it seems like a logical way to test whether the produced input is valid or not. As mentioned earlier, the input validator only needs to see the nextpass.in since that is what the submission would see and should always follow the same rules (stateless).

[evouga]

One additional remark: For many problems, it will be impossible to write a proper input validator. What I mean by that is that we can only check the format, but that is not sufficient to say that the input is valid... From the submissions' perspective, the input is only valid if it was derived from the submissions output, but checking that is impossible.

I agree, but "the input is derived from the submission output" doesn't need to be validated since that is always true by construction. It is still useful to check the syntax of the next pass input, as well as any semantics the problem author wishes to check (to make sure that there is no bug in the pass-to-pass transition logic).

[mzuenni]

I just noticed an issue/something that we should consider. Suppose you have a problem where you have to encode/decode something. Before this change, the Input validator could ensure that all test cases start with encode (which is very likely what you want). Since we know always run the input validator, and it is not aware of the pass, it can no longer ensure that there is no test case that accidentally looks like a submission output (more precisely like a nextpass.in). But in most cases, you probably need to avoid this since the nextpass.in is only valid fo a submission if it was derived from a submission.

[RagnarGrootKoerkamp]

Should we pass at least the pass number to the input validator as context? --pass <i>

[Matistjati]

The arguments to the input validator are already a little overloaded. Another slightly ugly solution would be to create a file in the working directory that indicates the current pass.

[mzuenni]

After thinking about this for quite some time, I think the best solution is to not run the input validator on nextpass.in.
I think it is better to either add extra code to the output validator or add a second program in the output_validator directory as well as a run script which runs them in the expected order.

I think there are multiple advantages of this approach:

• If the validation is done in the context of the output validator, it is clear that it is actually run. (Since the validation step is expected to succeed, tools could just decide to skip it, like DOMjudge does for input validation)
• it ensures that the validation can access all required information and gets access to the same persistent feedbackdir and by this it can also determine the current pass
• no changes to intend and behavior of input validators compared to normal problems
• I would expect that for many problems this is the easier approach