Clean up the details of static validators (#377)

* Clean up the details of static validators

* Add the score back in for static validation test cases
Updated the scoring section to take into account this score
Moved static validators to after output validators

* Point to maximum score inference

---------

Co-authored-by: Etienne Vouga <[email protected]>

Author: evouga

spec/2023-07-draft.md

@@ -74,8 +74,8 @@ File or Folder         | Required? | Described In
 `input_validators/`    | Yes       | [Input Validators](#input-validators)         | Programs that verifies correctness of the test data inputs
 `input_visualizer/`    | No        | [Input Visualizer](#input-visualizer)         | Scripts and documentation about how test case illustrations were generated
 `output_validator/`    | No        | [Output Validator](#output-validator)         | Custom program for judging solutions
-`output_visualizer/`   | No        | [Output Visualizer](#output-visualizer)       | Program to generate images illustrating submission output
 `static_validator/`    | No        | [Static Validator](#static-validator)         | Custom program for judging solutions with source files as input
+`output_visualizer/`   | No        | [Output Visualizer](#output-visualizer)       | Program to generate images illustrating submission output
 
 A minimal problem package must contain `problem.yaml`, a problem statement, a secret test case, an accepted judge solution, and an input validator.
 
@@ -1168,6 +1168,55 @@ It must not write to `score.txt`, `teammessage.txt`, or any other files in the f
 
 Compile or run-time errors in the visualizer are not judge errors. The return value and any data written by the visualizer to standard error or standard output are ignored.
 
+## Static Validator
+
+### Overview
+
+A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
+Optionally, the static validator may assign a score to the submission for each validation test case.
+By default there is no static validator.
+A static validator may be provided under the `static_validator` directory, similar to a custom output validator.
+
+### Static Validation Test Cases
+
+Each test group may define a static validation test case.
+It is an error to define static validation test cases without providing a static validator.
+A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
+If a map is specified, its allowed key are:
+- `args`, which maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case;
+- `score`, the maximum score of the static validation test case (see [Scoring Problems](#scoring-problems) for details).
+
+The `static_validation` key can also have the value of `false` meaning there is no static validation, or `true` meaning that static validation is enabled with no additional arguments and unspecified maximum score (to be determined by [maximum score inference](#maximum-score-inference)).
+
+It is an error to provide a static validator for `submit-answer` type problems, or to specify a `score` in a test group with `pass-fail` aggregation.
+
+### Invocation
+
+When invoked, the static validator will be passed at least three command line parameters.
+
+The validator should be possible to use as follows on the command line:
+```sh
+<static_validator_program> language entry_point feedback_dir [additional_arguments]
+```
+
+The meaning of the parameters listed above are:
+
+- language:
+  a string specifying the code of the language of the submission as shown in the [languages table](languages.md). A static validator must handle all of the programming languages specified in the `languages` key of `problem.yaml`.
+
+- entry_point:
+  a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.
+
+- feedback_dir:
+  a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
+  The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
+  so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.
+
+- additional_arguments:
+  in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.
+
+The static validator follows the semantics of an output validator for [reporting a judgment](#reporting-a-judgement).
+
 ## Verdict/Score Aggregation
 
 ### Pass-Fail Problems
@@ -1206,7 +1255,7 @@ Parent Group Maximum Score | Aggregation Type     | Default Maximum Score of Tes
 bounded value `M`          | `sum`                | `(M - S)/(A + T)`
 bounded value `M`          | `min`                | `M`
 
-where the group has `T` test cases, `A` subgroups without a provided `score`, and whose other subgroups have maximum scores that sum to `S`. 
+where the group has `T` non-static-validation test cases, `A` subgroups and static validation test cases without a provided `score`, and whose other subgroups and static validation test cases have maximum scores that sum to `S`. 
 This formula evenly distributes a group's leftover maximum points to its test cases and subgroups with unspecified maximum score. 
 It is a judge error if `S > M` for a group with bounded maximum score and `sum` aggregation.
 
@@ -1216,7 +1265,7 @@ Only test cases in test case groups with `sum` or `min` aggregation receive a sc
 
 The score of a failed test case is always 0.
 
-A custom output validator may produce a `score.txt` or `score_multiplier.txt` file for an accepted test case:
+A custom output validator or static validator may produce a `score.txt` or `score_multiplier.txt` file for an accepted test case:
 
 - for test cases with bounded maximum score, `score_multiplier.txt`, if produced, must contain a single floating-point number in the range `[0,1]`.
   The score of the test case is this number _multiplied_ by the test case maximum score. 
@@ -1225,14 +1274,14 @@ A custom output validator may produce a `score.txt` or `score_multiplier.txt` fi
 - for test cases with bounded maximum score, if no `score_multiplier.txt` or `score.txt` is produced, the test case score is its maximum score.
 - for test cases with unbounded maximum score, `score.txt` must be produced and must contain a non-negative floating-point number.
   The score of the test case is that number.
-
+  
 It is a judge error if:
-- an output validator accepts a test case in an unbounded group and does not produce a `score.txt`;
-- an output validator does not accept a test case, but does produce a `score.txt` or a `score_multiplier.txt`;
-- an output validator produces a `score_multiplier.txt` for a test case with unbounded maximum score;
-- an output validator produces both a `score.txt` and a `score_multiplier.txt` for a test case;
-- an output validator produces a `score.txt` or `score_multiplier.txt` for a test case in a group with `pass-fail` aggregation;
-- an output validator produces a `score.txt` or `score_multiplier.txt` with invalid contents.
+- an output or static validator accepts a test case in an unbounded group and does not produce a `score.txt`;
+- an output or static validator does not accept a test case, but does produce a `score.txt` or a `score_multiplier.txt`;
+- an output or static validator produces a `score_multiplier.txt` for a test case with unbounded maximum score;
+- an output or static validator produces both a `score.txt` and a `score_multiplier.txt` for a test case;
+- an output or static validator produces a `score.txt` or `score_multiplier.txt` for a test case in a group with `pass-fail` aggregation;
+- an output or static validator produces a `score.txt` or `score_multiplier.txt` with invalid contents.
 
 #### Scoring Test Groups
 
@@ -1260,52 +1309,3 @@ The paths of these required test cases or groups, relative to the `data` folder,
 The path of a group, relative to the `data/` folder, must come later lexicographically than the paths of all dependent test cases and groups.
 
 Each required group must be either `sample` or a subgroup of `secret` with `pass-fail` aggregation.
-
-## Static Validator
-
-### Overview
-
-A static validator is a program that is given the submission files as input and can analyze the contents to accept or reject the submission.
-Optionally, the static validator may assign a score to the submission.
-By default there is no static validator.
-A static validator may be provided under the `static_validator` directory, similar to a custom output validator.
-
-### Static Validation Test Cases
-
-Each test group may define a static validation test case.
-It is an error to define static validation test cases without providing a static validator.
-A static validation test case is defined within a group's `test_group.yaml` file by specifying the key `static_validation`.
-If a map is specified, it may have two keys `args`, and in the case of scoring test groups, `score`.
-The key `args` maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case.
-The key `score` maps to a float which represents both the maximum score achievable for the static validation test case and the default score assigned in case the static validator accepts the submission for that test case.
-The static validator can override this score by outputting a value to `score.txt` in the feedback directory, the same as an output validator.
-It can have the value of `false` meaning there is no static validation, `true` meaning that static validation is enabled with no score defined and no arguments.
-Aggregation is then applied to the test case in the same manner as other test cases.
-It is an error to assign a score in a `pass-fail` test group.
-It is an error to not assign a score in a `min` or `sum` test group.
-It is also an error to provide a static validator for `submit-answer` type problems.
-
-### Invocation
-
-When invoked, the static validator will be passed at least three command line parameters.
-
-The validator should be possible to use as follows on the command line:
-```sh
-<static_validator_program> language entry_point feedback_dir [additional_arguments]
-```
-
-The meaning of the parameters listed above are:
-
-- language:
-  a string specifying the code of the language of the submission as shown in the [languages table](languages.md).
-
-- entry_point:
-  a string specifying the entry point, that is a filename, class name, or some other identifier, which the static validator should know how to use depending on the language of the submission.
-
-- feedback_dir:
-  a string which specifies the name of a "feedback directory" in which the validator can produce "feedback files" in order to report additional information on the validation of the submission.
-  The feedback_dir must end with a path separator (typically '/' or '\\' depending on operating system),
-  so that simply appending a filename to feedback_dir gives the path to a file in the feedback directory.
-
-- additional_arguments:
-  in case the static validation test case specifies additional args, these are passed as additional arguments to the validator on the command line.