Validate test cases against provided schema

.github/workflows/validate_tests.yml

@@ -0,0 +1,20 @@
+name: Validate test data
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches: '**'
+
+jobs:
+  run_all:
+    name: Validate tests for specific schema version
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+      - name: Validate tests for schema v0.0.1
+        run: |
+            cd test
+            python3 validate.py --schema-dir ./schemas/v0-0-1/

test/.vscode/settings.json

@@ -0,0 +1,10 @@
+{
+    "json.schemas": [
+        {
+            "fileMatch": [
+                "tests/**/*.json"
+            ],
+            "url": "./schemas/v0-0-1/tests.schema.json"
+        }
+    ]
+}

test/README.md

@@ -1,37 +1,25 @@
-The files in this directory were originally copied from the [messageformat project](https://github.com/messageformat/messageformat/tree/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/__fixtures)
+The tests in the `./tests/` directory were originally copied from the [messageformat project](https://github.com/messageformat/messageformat/tree/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/__fixtures)
 and are here relicensed by their original author (Eemeli Aro) under the Unicode License.
 
 These test files are intended to be useful for testing multiple different message processors in different ways:
 
-- `syntax-errors.json` — An array of strings that should produce a Syntax Error when parsed.
-
-- `data-model-errors.json` - An object with string keys and arrays of strings as values,
-     where each key is the name of an error and its value is an array of strings that
-     should produce `error` when processed.
-     Error names are defined in ["MessageFormat 2.0 Errors"](../spec/errors.md) in the spec.
-
-- `test-core.json` — An array of test cases that do not depend on any registry definitions.
-  Each test may include some of the following fields:
-  - `src: string` (required) — The MF2 syntax source.
-  - `exp: string` (required) — The expected result of formatting the message to a string.
-  - `locale: string` — The locale to use for formatting. Defaults to 'en-US'.
-  - `params: Record<string, string | number | null | undefined>` — Parameters to pass in to the formatter for resolving external variables.
-  - `parts: object[]` — The expected result of formatting the message to parts.
-  - `cleanSrc: string` — A normalixed form of `src`, for testing stringifiers.
-  - `errors: { type: string }[]` — The runtime errors expected to be emitted when formatting the message.
-     If `errors` is either absent or empty, the message must be formatted without errors.
-  - `only: boolean` — Normally not set. A flag to use during development to only run one or more specific tests.
-
-- `test-function.json` — An object with string keys and arrays of test cases as values,
-  using the same definition as for `test-core.json`.
-  The keys each correspond to a function that is used in the tests.
-  Since the behavior of built-in formatters is implementation-specific,
-  the `exp` field should generally be omitted,
-  except for error cases.
-
-TypeScript `.d.ts` files are included for `test-core.json` and `test-function.json` with the above definition.
+- `syntax.json` — Test cases that do not depend on any registry definitions.
+
+- `syntax-errors.json` — Strings that should produce a Syntax Error when parsed.
+
+- `data-model-errors.json` - Strings that should produce Data Model Error when processed.
+  Error names are defined in ["MessageFormat 2.0 Errors"](../spec/errors.md) in the spec.
+
+- `functions/` — Test cases that correspond to built-in functions.
+  The behaviour of the built-in formatters is implementation-specific so the `exp` field is often
+  omitted and assertions are made on error cases.
 
 Some examples of test harnesses using these tests, from the source repository:
+
 - [CST parse/stringify tests](https://github.com/messageformat/messageformat/blob/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/cst/cst.test.ts)
 - [Data model stringify tests](https://github.com/messageformat/messageformat/blob/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/data-model/stringify.test.ts)
 - [Formatting tests](https://github.com/messageformat/messageformat/blob/11c95dab2b25db8454e49ff4daadb817e1d5b770/packages/mf2-messageformat/src/messageformat.test.ts)
+
+A JSON schema is included for the test files in this repository. [Semantic-versioned](https://semver.org/) schema files can be found in the [schemas folder](./schemas/).
+
+For users of Visual Studio Code, a [settings file](./.vscode/settings.json) is included that enables schema validation while editing the test files.