Rename testdata.yaml to test_group.yaml

Fixes #376.

Author: mpsijm

spec/2023-07-draft.md

@@ -396,7 +396,7 @@ All constant sequences in the following files will be replaced by the value of t
 - input and output validators
 - included code
 - example submissions
-- `testdata.yaml`
+- `test_group.yaml`
 
 Note that constants are also available in LaTeX problem statements via the dedicated command `\constant{name}`.
 
@@ -495,7 +495,7 @@ For example, the files `secret/test.in` and `secret/test.ans` are associated wit
 The existence of the `.in` file declares the existence of the test case.
 If the test case exists, then an associated `.ans` file must exist while the others are optional.
 If the test case does not exist, then the other files must not exist.
-Note that a test case must not be named `*/testdata`, since `testdata.yaml` would then be configuration for both the test case and test group.
+Note that a test case must not be named `*/test_group`, since `test_group.yaml` would then be configuration for both the test case and test group.
 The table below summarizes the supported test data:
 
 Extension                       | Described In         | Summary
@@ -547,10 +547,10 @@ Any unknown keys should be treated as an error.
 
 Key                   | Type                | Default
 --------------------- | ------------------- | -------
-args                  | Sequence of strings | Inherited from [`testdata.yaml`](#test-data-groups), which defaults to empty sequence
-output_validator_args | Sequence of strings | Inherited from [`testdata.yaml`](#test-data-groups), which defaults to empty sequence
-input_validator_args  | Sequence of strings or map of strings to sequences of strings | Inherited from [`testdata.yaml`](#test-data-groups), which defaults to empty sequence
-full_feedback         | Boolean             | Inherited from [`testdata.yaml`](#test-data-groups), which defaults to `false` in `secret` and `true` in `sample`
+args                  | Sequence of strings | Inherited from [`test_group.yaml`](#test-data-groups), which defaults to empty sequence
+output_validator_args | Sequence of strings | Inherited from [`test_group.yaml`](#test-data-groups), which defaults to empty sequence
+input_validator_args  | Sequence of strings or map of strings to sequences of strings | Inherited from [`test_group.yaml`](#test-data-groups), which defaults to empty sequence
+full_feedback         | Boolean             | Inherited from [`test_group.yaml`](#test-data-groups), which defaults to `false` in `secret` and `true` in `sample`
 hint                  | String              | 
 description           | String              | 
 
@@ -586,12 +586,12 @@ If a specific order is desired, a numbered prefix such as `00`, `01`, `02`, `03`
 A subgroup must not have the same name as a test case in the same directory.
 For example, if the file `data/secret/huge.in` exists then the directory `data/secret/huge/` must not, and vice versa.
 
-In each test data group, a YAML file `testdata.yaml` may be placed to specify how the result of the test data group should be computed.
-Some of the keys and their associated values will be inherited from the `testdata.yaml` in the closest ancestor group from the test case to the root `data` directory that has one.
-Others must be explicitly defined in the group's `testdata.yaml` file — otherwise they are set to the default values.
-If there is no `testdata.yaml` file in the root `data` group, one is implicitly added with the default values.
+In each test data group, a YAML file `test_group.yaml` may be placed to specify how the result of the test data group should be computed.
+Some of the keys and their associated values will be inherited from the `test_group.yaml` in the closest ancestor group from the test case to the root `data` directory that has one.
+Others must be explicitly defined in the group's `test_group.yaml` file — otherwise they are set to the default values.
+If there is no `test_group.yaml` file in the root `data` group, one is implicitly added with the default values.
 
-The format of `testdata.yaml` is as follows:
+The format of `test_group.yaml` is as follows:
 
 Key                     | Type                                | Default                                 | Inheritance       | Comments
 ----------------------- | ----------------------------------- | --------------------------------------- | ----------------- | --------
@@ -627,7 +627,7 @@ In particular, for an existing feedback directory `dir`,
 <output_validator_program> tc.in tc.ans dir [arguments] < tc.out # MUST FAIL
 ```
 
-The directory `invalid_output` must be organized into a tree-like structure similar to `secret` and may contain arguments in `testdata.yaml` files that are passed to the validators.
+The directory `invalid_output` must be organized into a tree-like structure similar to `secret` and may contain arguments in `test_group.yaml` files that are passed to the validators.
 
 ### Valid Output
 
@@ -644,7 +644,7 @@ In particular, for an existing feedback directory `dir`,
 <output_validator_program> tc.in tc.ans dir [arguments] < tc.out # MUST PASS
 ```
 
-The directory `invalid_output` must be organized into a tree-like structure similar to `secret` and may contain arguments in `testdata.yaml` files that are passed to the validators.
+The directory `invalid_output` must be organized into a tree-like structure similar to `secret` and may contain arguments in `test_group.yaml` files that are passed to the validators.
 
 ### Samples
 
@@ -723,7 +723,7 @@ Otherwise the `.ans` is used as the `team_output`.
 Files in `data/sample/download` are never validated, 
 although tooling may warn when there are inconsistencies between `download`, `statement`, and `data/sample/` itself.
 
-Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/testdata.yaml`. 
+Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`.
 These arguments are used when validating test cases in both `/data/sample` and `/data/sample/statement`.
 
 ## Generators
@@ -1181,7 +1181,7 @@ The goal of each submission is to maximize this score.
 Only the `secret` group and its subgroups are scored.
 
 Given a submission, scores are determined for test cases, test groups, and the submission itself (which is the score of the `secret` group).
-The scoring behavior is configured for each test data group by the following arguments in the `scoring` dictionary of its `testdata.yaml`:
+The scoring behavior is configured for each test data group by the following arguments in the `scoring` dictionary of its `test_group.yaml`:
 
 Key           | Type                          | Description
 ------------- | ----------------------------- | -----------
@@ -1274,7 +1274,7 @@ A static validator may be provided under the `static_validator` directory, simil
 
 Each test group may define a static validation test case.
 It is an error to define static validation test cases without providing a static validator.
-A static validation test case is defined within a group's `testdata.yaml` file by specifying the key `static_validation`.
+A static validation test case is defined within a group's `test_group.yaml` file by specifying the key `static_validation`.
 If a map is specified, it may have two keys `args`, and in the case of scoring test groups, `score`.
 The key `args` maps to a string which represents the additional arguments passed to the static validator in this group's static validation test case.
 The key `score` maps to a float which represents both the maximum score achievable for the static validation test case and the default score assigned in case the static validator accepts the submission for that test case.

spec/changelog.md

@@ -34,6 +34,7 @@ sort: 1
 - Add `/submissions/rejected` directory.
 - Clarify working directories for submissions and validators.
 - Add `allow_writing_files` to `problem.yaml`.
+- Rename `testdata.yaml` to `test_group.yaml`.
 
 ## Legacy version (changes since beginning 2021)
 

support/schemas/testdata.cue renamed to support/schemas/test_group.cue

@@ -1,13 +1,13 @@
 package problem_package
 
-#testdata_settings: {
+#test_group_settings: {
 	input_validator_flags?:  *"" | string | {[string]: string}
 	output_validator_flags?: *"" | string
-        grading?: {
+	grading?: {
 		score?:       number
 		max_score?:   number
 		aggregation?: "sum" | "min"
 	}
 }
 
-#testdata_settings
+#test_group_settings