Skip to content

Clarify dependent groups #487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Clarify dependent groups #487

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 14 additions & 16 deletions spec/2023-07-draft.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,7 +540,8 @@ The `secret` directory must exist, and contain either some test cases, or some [

All files and directories associated with a single test case have the same base name with varying extensions.
Here, base name is defined to be the relative path from the `data` directory to the test case input file, without extensions.
For example, the files `secret/test.in` and `secret/test.ans` are associated with the same test case that has the base name `secret/test`.
This is the name of the test case.
For example, the files `secret/test.in` and `secret/test.ans` are associated with the same test case that has the name `secret/test`.
The existence of the `.in` file declares the existence of the test case.
If the test case exists, then an associated `.ans` file must exist while the others are optional.
If the test case does not exist, then the other files must not exist.
Expand All @@ -561,7 +562,7 @@ as well as the `args` sequence in the `.yaml` file, then the input of the two te
This means that for any two test cases, if their input, output validator arguments and the contents of their `.ans` files are equivalent, then the test cases are equivalent.
The assumption of determinism means that a judge system could choose to reuse the result of a previous run, or to re-run the equivalent test case.

Test cases and [test data groups](#test-data-groups) will be used in lexicographical order on base name.
Test cases and [test data groups](#test-data-groups) will be used in lexicographical order on test case or test group name.
It is good practice to use a numbered prefix such as `00`, `01`, `02`, `03`, and so on, to get the desired order of test cases, while keeping the file names descriptive.
Remember that the numbered prefixes should be zero padded to the same length to get the expected lexicographical order.

Expand All @@ -576,7 +577,7 @@ Key | Type | Default
`max_score` | Integer or `unbounded` | 100 in `secret`, otherwise `unbounded` | The maximum possible score of the test data group. Must be a non-negative integer or `unbounded`. This key is only permitted for the `secret` group and its subgroups.
`score_aggregation` | `pass-fail`, `sum`, or `min` | `sum` in `secret`, otherwise `pass-fail` | How the score is determined based on the scores of the contained groups or test cases. [See Result Aggregation](#result-aggregation). This key is only permitted for the `secret` group and its subgroups.
`static_validation_score` | Integer or `pass-fail` | | The maximum score of the static validation test case, or `pass-fail`. See [Static Validator](#static-validator).
`require_pass` | String or sequence of strings | empty sequence | Other test data groups whose test cases a submission must pass in order to receive a score for this test group. [See Result Aggregation](#result-aggregation). This key is only permitted for the `secret` group and its subgroups.
`require_pass` | String or sequence of strings | empty sequence | Names of other test data groups, or `sample`, whose test cases a submission must pass in order to receive a score for this test group. [See Result Aggregation](#result-aggregation). This key is only permitted for the `secret` group and its subgroups.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This key is only permitted for the `secret` group and its subgroups.

This is not correct. It is technically also allowed for sample, as mentioned further down under "Group Dependencies".

Also it should not refer to Result Aggregation, and refer directly to Group Dependencies.

Suggested change
`require_pass` | String or sequence of strings | empty sequence | Names of other test data groups, or `sample`, whose test cases a submission must pass in order to receive a score for this test group. [See Result Aggregation](#result-aggregation). This key is only permitted for the `secret` group and its subgroups.
`require_pass` | String or sequence of strings | empty sequence | Names of other test data groups, or `sample`, whose test cases a submission must pass in order to receive a score for this test group. See [Group Dependencies](#group-dependencies).

`args` | Sequence of strings | empty sequence | See [Test Case Configuration](#test-case-configuration).
`input_validator_args` | Sequence of strings or map of strings to sequences of strings | empty sequence | See [Test Case Configuration](#test-case-configuration).
`static_validator_args` | Sequence of strings | empty sequence | See [Static Validator](#static-validator).
Expand All @@ -590,6 +591,8 @@ Key | Type | Default
The secret data may be subdivided into test data groups.
Every subdirectory of `data/secret/` is a test data group and may contain a `test_group.yaml` configuration file.
`data/secret` can only have test data groups *or* test cases, never both.
The name of a test data group is the path under secret.
For example, the test data group defined by the directory `data/secret/group1` has the name `secret/group1`.
Comment on lines +594 to +595
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move the lines

The name of a test data group is the path under secret.
For example, the test data group defined by the directory `data/secret/group1` has the name `secret/group1`.

under

That is, if there are any directories under `data/secret/` there must not be any `.in` files directly in `data/secret/` and vice versa.

That is, if there are any directories under `data/secret/` there must not be any `.in` files directly in `data/secret/` and vice versa.

The test groups themselves can contain directories, but not further groups.
Expand All @@ -604,7 +607,7 @@ Each test data group must contain at least one test case, or a static validation

Each test case can supply input via standard input, command-line arguments, and/or the file system.
These options are not exclusive.
For a test case with base name `test`, the file `test.in` is piped to the submission as standard input.
For a test case with name `test`, the file `test.in` is piped to the submission as standard input.
The submission will be run with the `args` sequence defined in the `test.yaml` file as command-line arguments.
Note that usually the submission's entry point, whether it be a binary or an interpreted file, will be the absolute first command line argument.
However, there exist languages, such as Java, where there is no initial command line argument representing the entry point.
Expand All @@ -616,7 +619,6 @@ possibly overwriting the compiled submission file or included data in the case o
### Test Case Configuration

One YAML file with additional configuration may be provided per test case.
The file must share the base name of the associated test case.

The allowed keys are defined as follows.
Keys are optional unless explicitly stated.
Expand Down Expand Up @@ -656,7 +658,6 @@ For each test case:

An illustration provides a visualization of the associated test case, meant for the judges.
At most one illustration file may be provided per test case.
The file must share the base name of the associated test case.
The supported file extensions are `.png`, `.jpg`, `jpeg`, and `.svg`.

### Invalid Test Cases
Expand Down Expand Up @@ -1352,14 +1353,11 @@ The submission score is the score of the `secret` group.

It is a judge error if the score of any group or subgroup exceeds its `max_score`.

#### Required Dependent Groups
#### Group Dependencies

A group may specify that it depends on some other test data groups.
Each required group must be either `sample` or have `pass-fail` aggregation.
The dependent group will only be run if the group being depended on receives an accepted verdict for all test cases in the group.
If the dependent group is not run, the group score is 0.

The paths of these required groups, relative to the `data` folder, are listed under the `require_pass` key.
A path consists of zero or more directory names followed by a directory or file name, with a `/` character separating consecutive names. Each name must conform to the [general requirements](#general-requirements) on directory and file names and the specified test data group must exist.

The path of a group, relative to the `data/` folder, must come later lexicographically than the paths of all test cases and groups it depends on.
A test data group, or `secret` may specify that it depends on some other test data groups, or `sample` by setting `require_pass`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, secret can only depend on sample, right? Maybe that should be specified.
And sample cannot depend on anything else, since it is the first group. But it is allowed to have a require_pass key.

Suggested change
A test data group, or `secret` may specify that it depends on some other test data groups, or `sample` by setting `require_pass`.
A test data group, or `secret` may specify that it depends on some other test data groups. `secret` may only depend on `sample`. `sample` cannot depend on any other group, but it may still have a `require_pass` key.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct, and those are good points.

Each such test data group must have `pass-fail` aggregation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got a bit confused if it was the groups that depend on others, or the groups dependent on that needs a pass-fail aggregation.

We should specify that this only applies to the test data groups being dependent on.

Suggested change
Each such test data group must have `pass-fail` aggregation.
Each group mentioned under `require_pass` must have `pass-fail` aggregation.

The group specifying `require_pass` will only be run if the group being referred to receives an accepted verdict for all test cases in the group.
If the group is not run, the group score is 0, and it did not receive an accepted verdict for all test cases in the group.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and it did not receive an accepted verdict for all test cases in the group.

A bit confusing. I assume this is to clarify that even if a group $A$ has max score 0 and is not run, the other groups that depends on $A$ will also not be run?

The `require_pass` is set to the name of a single test data group, or `sample`, or a sequence of such names, or `sample`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit confusing thatsample is mentioned twice here.

Suggestion:

Suggested change
The `require_pass` is set to the name of a single test data group, or `sample`, or a sequence of such names, or `sample`.
The `require_pass` is set to the name of a single test data group, or `sample`, or a sequence of such names.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yours read a lot better, but technically sample is not defined to be a "name". That said... I'm inclined to merge your change.

Anybody thinks differently?

All groups listed in `require_pass` must come lexicographically earlier than the group it is specified for.