Skip to content

Commit 5cc1eb3

Browse files
committed
Reorder validator and visualizer sections
1 parent b2f93a3 commit 5cc1eb3

File tree

1 file changed

+54
-54
lines changed

1 file changed

+54
-54
lines changed

spec/2023-07-draft.md

Lines changed: 54 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -72,9 +72,9 @@ File or Folder | Required? | Described In
7272
`include/` | No | [Included Files](#included-files) | Files appended to all submitted solutions
7373
`submissions/` | Yes | [Example Submissions](#example-submissions) | Correct and incorrect judge solutions of the problem
7474
`input_validators/` | Yes | [Input Validators](#input-validators) | Programs that verifies correctness of the test data inputs
75-
`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated
76-
`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions
7775
`static_validator/` | No | [Static Validator](#static-validator) | Custom program for judging solutions with source files as input
76+
`output_validator/` | No | [Output Validator](#output-validator) | Custom program for judging solutions
77+
`input_visualizer/` | No | [Input Visualizer](#input-visualizer) | Scripts and documentation about how test case illustrations were generated
7878
`output_visualizer/` | No | [Output Visualizer](#output-visualizer) | Program to generate images illustrating submission output
7979

8080
A minimal problem package must contain `problem.yaml`, a problem statement, a secret test case, an accepted judge solution, and an input validator.
@@ -949,10 +949,54 @@ Any other exit code means that the input file could not be confirmed as valid.
949949
The validator **must not** read any files outside those defined in the Invocation section.
950950
Its result **must** depend only on these files and the arguments.
951951

952-
## Input Visualizer
952+
## Static Validator
953953

954-
If a tool was used to automate creating test case illustration annotations,
955-
it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`.
954+
### Overview
955+
956+
A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
957+
Optionally, the static validator may assign a score to the submission for each validation test case.
958+
By default there is no static validator.
959+
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.
960+
961+
### Static Validation Test Cases
962+
963+
Each test group may define a static validation test case.
964+
It is an error to define static validation test cases without providing a static validator.
965+
A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
966+
If a map is specified, its allowed key are:
967+
- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
968+
- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).
969+
970+
The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).
971+
972+
It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.
973+
974+
### Invocation
975+
976+
When invoked, the static validator will be passed at least three command line parameters.
977+
978+
The validator should be possible to use as follows on the command line:
979+
```sh
980+
<static_validator_program> language entry_point feedback_dir [additional_arguments]
981+
```
982+
983+
The meaning of the parameters listed above are:
984+
985+
- language:
986+
a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.
987+
988+
- entry_point:
989+
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.
990+
991+
- feedback_dir:
992+
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
993+
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
994+
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.
995+
996+
- additional_arguments:
997+
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.
998+
999+
The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).
9561000

9571001
## Output Validator
9581002

@@ -1147,6 +1191,11 @@ Almost all test cases failed — are you even trying to solve the problem?
11471191
A validator program is allowed to write any kind of debug information to its standard error pipe.
11481192
This information may be displayed to the user upon invocation of the validator.
11491193

1194+
## Input Visualizer
1195+
1196+
If a tool was used to automate creating test case illustration annotations,
1197+
it is recommended to include the input visualizer source code in the directory `input_visualizer/` along with invocation instructions in a file such as `input_visualizer/README.txt`.
1198+
11501199
## Output Visualizer
11511200

11521201
An output visualizer is an optional [program](#programs) that is run after every invocation of the output validator in order to generate images illustrating the submission output.
@@ -1159,55 +1208,6 @@ It must not write to `score.txt`, `teammessage.txt`, or any other files in the f
11591208

11601209
Compile or run-time errors in the visualizer are not judge errors. The return value and any data written by the visualizer to standard error or standard output are ignored.
11611210

1162-
## Static Validator
1163-
1164-
### Overview
1165-
1166-
A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
1167-
Optionally, the static validator may assign a score to the submission for each validation test case.
1168-
By default there is no static validator.
1169-
A static validator may be provided under the `static_validator` directory, similar to a custom output validator.
1170-
1171-
### Static Validation Test Cases
1172-
1173-
Each test group may define a static validation test case.
1174-
It is an error to define static validation test cases without providing a static validator.
1175-
A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
1176-
If a map is specified, its allowed key are:
1177-
- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
1178-
- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).
1179-
1180-
The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).
1181-
1182-
It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.
1183-
1184-
### Invocation
1185-
1186-
When invoked, the static validator will be passed at least three command line parameters.
1187-
1188-
The validator should be possible to use as follows on the command line:
1189-
```sh
1190-
<static_validator_program> language entry_point feedback_dir [additional_arguments]
1191-
```
1192-
1193-
The meaning of the parameters listed above are:
1194-
1195-
- language:
1196-
a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.
1197-
1198-
- entry_point:
1199-
a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.
1200-
1201-
- feedback_dir:
1202-
a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
1203-
The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
1204-
so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.
1205-
1206-
- additional_arguments:
1207-
in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.
1208-
1209-
The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).
1210-
12111211
## Verdict/Score Aggregation
12121212

12131213
### Pass-Fail Problems

0 commit comments

Comments
 (0)