Allowing `pass-fail` problems to have both test data groups and test cases in `secret/`

[hairez]

To clarify, I would like to allow the following to be valid for pass-fail problems:

└── secret
    ├── dir1
    │   └──A.in
    └── B.in

Summarizing my understanding of discussion from #480 :

• The current formulation is The secret directory must exist, and contain either some test cases, or some test data groups, but not both.

• The PR Group and test case names cannot collide #492 has allowed directories and test cases with the same name to coexist at any level but secret.

• @RagnarGrootKoerkamp mentioned that it would be nice if this was allowed in secret as well for pass-fail problems.

• Some discussion regarding that there should not be any test data groups and only test data directories for pass-fail has been mentioned in Restricting depth of test data tree #410.

• Argument against this is that one might want to give the submission feedback for a whole group of test cases (Restricting depth of test data tree #410 (comment)). But I argue that submission feedback could be tied to a directory of test cases instead.

My suggestion is to allow test cases and directories to coexist in secret/ for pass-fail problems.

Any other opinions, or points that I have missed?

[Matistjati]

I don't really see the need to allow both test cases and directories? What do you lose by having either only testcases, or only test groups?

[hairez]

Ah right, you can always just put the test cases that are not in a directory, into some directory other:

└── secret
    ├── dir1
    │   └──A.in
    └── other
        └──B.in

I guess the only issue with this forces the test cases without directory to be in a consecutive interval when ordering the test cases, but even that can be solved by creating separate directories.

I’m also unsure whether it’s better for the person creating the test data to have to manually create an extra folder other, or if the package should automatically handle cases where test files are placed outside of directories.

[niemela]

Ah right, you can always just put the test cases that are not in a directory, into some directory other:

No, this is not equivalent. Note that groups are (typically) externally visible, while "organizational directories" are not.

As a slightly related side note, I think we should write something about that results are expected (?) (appropriate (?)) for groups. (I don't know what wording is good).

[hairez]

I was imagining that "groups" for pass-fail just gets treated as directories automatically, but maybe that is something to be discussed here as well.

[niemela]

I was imagining that "groups" for pass-fail just gets treated as directories automatically, but maybe that is something to be discussed here as well.

That might solve @RagnarGrootKoerkamp's issue? Should we just say that there can't be any test data groups for pass-fail problems? I.e. any directories are just "organizational".

@Matistjati is there any (reasonable) reason we need atest_data.yaml and/or groups for pass-fail?

[RagnarGrootKoerkamp]

Even for pass-fail, one could still have test_data.yaml with different output_validator_flags for subsets of the data, right?

[niemela]

Even for pass-fail, one could still have test_data.yaml with different output_validator_flags for subsets of the data, right?

Currently, absolutely!

That was basically what I was asking, and that's a good answer, so we do want to keep test data groups for pass-fail then.

[evouga]

Given that there is no good way to distinguish test groups from organizational directories, I am in favor of the status quo:

• subdirectories of secret are test groups; deeper subdirectories are organizational;
• secret can either have loose test cases, or test groups, but not both.

There are good reasons to forbid a mix of test groups and test cases in scoring problems (a mix significantly complicates aggregation), so it seems cleanest to me to consistently forbid a mix in pass-fail problems as well. The annoyance of putting loose test cases in an other directory is IMO quite minor.

I would also be in favor of a more substantial change that does clearly distinguish test groups from organizational directories, but that ship has probably sailed for the 2025/9 spec.

[niemela]

I would also be in favor of a more substantial change that does clearly distinguish test groups from organizational directories, but that ship has probably sailed for the 2025/9 spec.

Maybe, I would curious to hear what method you were thinking.

The one I'm pondering is making test_group.yaml mandatory, i.e. it's existence is what specified that it us a group.