rewrote statement sample section (#375)

* rewrote statement sample section

* Update 2023-07-draft.md

wording changes

* Update 2023-07-draft.md

* Update 2023-07-draft.md

* change wording

* add .out

* Update spec/2023-07-draft.md

Co-authored-by: Fredrik Niemelä <[email protected]>

* Update spec/2023-07-draft.md

Co-authored-by: Fredrik Niemelä <[email protected]>

* Update spec/2023-07-draft.md

Co-authored-by: Fredrik Niemelä <[email protected]>

* also validate .ans files if not overwritten

* Update spec/2023-07-draft.md

Co-authored-by: Fredrik Niemelä <[email protected]>

* Update spec/2023-07-draft.md

Co-authored-by: Fredrik Niemelä <[email protected]>

* mention directories

* Update spec/2023-07-draft.md

---------

Co-authored-by: Fredrik Niemelä <[email protected]>

Author: mzuenni

spec/2023-07-draft.md

@@ -655,8 +655,7 @@ Sample test cases can be used in three places:
 
 By default the sample data for all three cases is taken from the `.in` and `.ans` file pairs under `data/sample`.
 Some problems require (slightly) different data in each of these cases.
-We allow customizing which data is used for each purpose using the subdirectories `data/sample/statement` and `data/sample/download`.
-In this case it is recommended to symlink identical files from these subdirectories to those in `data/sample`.
+We allow customizing which data is used for each purpose with the additional extensions `.statement` and `.download`.
 
 #### Samples For Judging Team Submissions
 
@@ -670,9 +669,10 @@ It may be missing (for problems with no samples) or empty.
 #### Samples Shown in the Problem Statement
 
 By default, the `.in` and `.ans` pairs from `data/sample` are shown in the problem statement.
-This behavior can be customized by creating the `data/sample/statement` directory.
-If this directory exists, its contents replaces that of `data/sample` for purposes of the problem statement.
-Interactive problems with sample test cases should almost always make use of this directory, since the `.in` and `.ans` pairs do not meaningfully capture how users are expected to interact with the output validator in such problems.
+If a `.out` file exists the `.out` file is shown instead of the `.ans` file in the problem statement.
+This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`.
+If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension -- for purposes of the problem statement.
+Note that it is an error to provide both a `.out` and a `.ans.statement` file.
 
 ##### Interactive Problems
 
@@ -681,51 +681,48 @@ The validator gets access to the `.in` and `.ans` files for each test case and c
 Standard in of the output validator corresponds to standard out of the submission, and standard out of the output validator corresponds to standard in of the submission.
 Therefore, the output validator can control what information from the `.in` and `.ans` files is provided to the submission.
 
-For interactive problems, sample data in `data/sample/statement` must have one of the following forms:
-- An `.in` and `.out` file, which are both shown, and neither of which is validated.
-- An `.interaction` file that contains lines starting with `<` and `>`,
-  containing an interaction log with output from the output validator starting with `<` and output from the submission starting with `>`.
+For interactive problems displaying two files is typically not meaningfully to capture how users are expected to interact with the output validator.
+Therefore, it is advised to instead provide samples for the problem statement in the form of a `.interaction` file.
+This file contains lines starting with `<` and `>`, containing an interaction log with output from the output validator starting with `<` and output from the submission starting with `>`.
 
-Note that an `.interaction` file does **not** need to be accompanied by corresponding `.in` and `.out` files. If both an `.interaction` file and corresponding `.in` and `.out` files are present, only the interaction log in the `.interaction` file is shown in the problem statement.
+Note that if you are using a `.interaction` file you must not provide a `.in.statement`, `.ans.statement`, or `.out` file.
 
-If you want to make testing tools available for download, you can use [attachments](#attachments).
+##### Multi-Pass Problems
 
-##### Other Problem Types
+Multi-pass problems require a custom output validator, which interacts with the submission see [multi-pass](#multi-pass-validation).
 
-For each test case, `data/sample/statement` must contain one of the following:
-- An `.in` and `.ans` file, which are both shown.
-- An `.in`, `.ans`, and `.out` file, of which the `.in` and `.out` file are shown, and the `.ans` file is used for validation (see below).
+For multi-pass problems displaying two files is typically not meaningfully to capture how users are expected to interact with the output validator.
+Therefore, it is advised to instead provide samples for the problem statement in the form of a `.interaction` file.
+This file contains lines starting with `<` and `>`, like for interactive problems.
+Passes are separated by a line containing `---` (three dashes).
+When the problem is not interactive, simply start each pass by a number of lines starting with `<`, containing the sample input, followed by some lines starting with `>`, containing the sample output.
 
+Note that if you are using a `.interaction` file you must not provide a `.in.statement`, `.ans.statement`, or `.out` file.
 
 #### Samples Available for Download
 
-When `data/sample/download` exists, the files in there are available for download.
-If this directory does not exist, the contents of `data/sample/statement` are available for download,
-with any `.out` files renamed to `.ans` (replacing existing `.ans` files) and all `.interaction` files removed.
-When that also does not exist, the test cases in `data/sample` are available for download.
+By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download.
+Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download.
+This behavior can be further customized by providing files with the extension `in.download` or `ans.download`.
+If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download.
+Additionally, any other file or directory with the extension `.download` is also available for download (without the `.download` extension).
 
-The `data/sample/download` directory may contain anything, and none of its contents are validated. 
-We recommend placing test-case-specific files in this directory, and using [attachments](#attachments) for testing tools and other downloads not associated with any particular test case.
-Testing tools may warn when test cases that are listed in `data/sample` or `data/sample/statement` do not appear in `data/sample/download`,
-or when `data/sample/download` contains test cases that do not appear in one of the two other locations.
+If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments).
 
 #### Validation
 
-All `data/sample/*.in` files are input-validated. For problems that are not interactive, all `data/sample/statement/*.in` are also input-validated.
+All `data/sample/*.in` files are input-validated.
+For non interactive and non multi-pass problems the `.out` files must pass the output validator.
+For non interactive and non multi-pass problems the `.ans` files must pass the output validator if they are not overriden in any way, i.e., if they are shown in the statement.
+All other files are not validated in any way.
 
-When `data/sample/statement/` does not exist, all test cases in `data/sample` must be accepted by the output validator, with the `.ans` file used as both the `answer_file` and `team_output`.
- 
-When `data/sample/statement/` does exists, and the problem is not interactive, all test cases in `data/sample/statement/` must be accepted by the output validator. 
-The `.ans` is used as the `answer_file`. 
-The `.out` file is used as the `team_output`, if it exists. 
-Otherwise the `.ans` is used as the `team_output`. 
-
-Files in `data/sample/download` are never validated, 
-although tooling may warn when there are inconsistencies between `download`, `statement`, and `data/sample/` itself.
+Note that `.ans.statement` and `.out` can both be used to change what is shown in the statement.
+However, since only the `.out` files are validated it is advised to use these if possible.
 
 Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`.
 These arguments are used when validating test cases in both `/data/sample` and `/data/sample/statement`.
 
+
 ## Generators
 
 If any generator scripts were used to automate writing test cases, 
@@ -1129,12 +1126,6 @@ It is a judge error to run more passes than specified by the `limits.validation_
 All other files inside the feedback directory are guaranteed to persist between passes.
 In particular, the validator should only append text to the `judgemessage.txt` to provide combined feedback for all passes.
 
-Samples for multi-pass problems must be provided in a `.interaction` file,
-like for interactive problems. Passes are separated by a line containing `---` (three dashes).
-When the problem is not interactive,
-simply start each pass by a number of lines starting with `<`, containing the sample input,
-followed by some lines starting with `>`, containing the sample answer.
-
 #### Examples
 
 An example of a `judgemessage.txt` file: