Skip to content

Commit e9f5400

Browse files
gregsdennisgregsdennis
committed
initial run at creating output tests
1 parent 716b95d commit e9f5400

File tree

7 files changed

+287
-0
lines changed

7 files changed

+287
-0
lines changed

‎output-tests/README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
These tests are intended to validate that implementations are correctly generating output in accordance with the specification.
2+
3+
Output was initially specified with draft 2019-09. It remained unchanged for draft 2020-12 <!-- do we need explicit tests for 2002-12? -->, but will receive an update with the next release.
4+
5+
## Organization
6+
7+
The tests are organized into two categories: content and structure.
8+
9+
Content tests verify that the keywords are producing the correct annotations. Since there are no requirements on the content of error messages, there's not much that can be verified for them, if they're even generated. These tests need to extensively cover the annotation behaviors of each keyword. The expected output format for these tests is `basic`.
10+
11+
Structure tests verify that the structures of the various formats (i.e. `flag`, `basic`, `detailed`, `verbose`) are correct. These tests don't need to cover each keyword; rather they need to sufficiently cover the various aspects of building the output structures by using whatever keywords are necessary to do so.
12+
13+
## Test Files
14+
15+
The content of a test file is the same as the validation tests in `tests/`, however an `output` property has been added to each test case.
16+
17+
The `output` property itself has a property for each of the output formats where the value is a schema that will successfully validate for compliant output. For the content tests, only `basic` needs to be present.
18+
19+
## Contributing
20+
21+
Of course, first and foremost, follow the [Contributing guide](/CONTRIBUTING.md).
22+
23+
When writing test cases, try to keep output validation schemas targeted to verify a single requirement. Where possible (and where it makes sense), create multiple tests to cover multiple requirements. This will help keep the output validation schemas small and increase readability. (It also increases your test count. 😉)
24+
25+
For the content tests, there is also a _general.json_ file that contains tests that do not necessarily pertain to any single keyword.
26+
<!-- This general.json file may be added to the structure tests later, but I haven't gotten to them yet, so I don't know. -->

‎output-tests/content/draft-next/general.json

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
[
2+
{
3+
"description": "failed validation produces no annotations",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/next/schema",
6+
"$id": "https://json-schema.org/tests/content/draft-next/general/1",
7+
"type": "string",
8+
"readOnly": true
9+
},
10+
"tests": [
11+
{
12+
"description": "dropped annotations MAY appear in droppedAnnotations",
13+
"data": 1,
14+
"output": {
15+
"basic": {
16+
"type": "object",
17+
"properties": {
18+
"details": {
19+
"contains": {
20+
"type": "object",
21+
"properties": {
22+
"evaluationPath": {"const": ""},
23+
"schemaLocation": {"const": "https://json-schema.org/tests/content/draft-next/general/1"},
24+
"instanceLocation": {"const": ""},
25+
"annotations": false,
26+
"droppedAnnotations": {
27+
"properties": {
28+
"readOnly": {"const": true}
29+
},
30+
"required": ["readOnly"]
31+
}
32+
},
33+
"required": ["evaluationPath", "schemaLocation", "instanceLocation"]
34+
}
35+
}
36+
},
37+
"required": ["details"]
38+
}
39+
}
40+
}
41+
]
42+
}
43+
]

‎output-tests/content/draft-next/readOnly.json

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
[
2+
{
3+
"description": "readOnly generates its value as an annotation",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/next/schema",
6+
"$id": "https://json-schema.org/tests/content/draft-next/readOnly/1",
7+
"readOnly": true
8+
},
9+
"tests": [
10+
{
11+
"description": "readOnly is true",
12+
"data": 1,
13+
"output": {
14+
"basic": {
15+
"type": "object",
16+
"properties": {
17+
"details": {
18+
"contains": {
19+
"type": "object",
20+
"properties": {
21+
"evaluationPath": {"const": ""},
22+
"schemaLocation": {"const": "https://json-schema.org/tests/content/draft-next/readOnly/1"},
23+
"instanceLocation": {"const": ""},
24+
"annotations": {
25+
"properties": {
26+
"readOnly": {"const": true}
27+
},
28+
"required": ["readOnly"]
29+
}
30+
},
31+
"required": ["evaluationPath", "schemaLocation", "instanceLocation", "annotations"]
32+
}
33+
}
34+
},
35+
"required": ["details"]
36+
}
37+
}
38+
}
39+
]
40+
}
41+
]

‎output-tests/content/draft-next/type.json

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
[
2+
{
3+
"description": "incorrect type",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/next/schema",
6+
"$id": "https://json-schema.org/tests/content/draft-next/type/1",
7+
"type": "string"
8+
},
9+
"tests": [
10+
{
11+
"description": "incorrect type may be reported",
12+
"data": 1,
13+
"output": {
14+
"basic": {
15+
"type": "object",
16+
"properties": {
17+
"details": {
18+
"contains": {
19+
"type": "object",
20+
"properties": {
21+
"evaluationPath": {"const": ""},
22+
"schemaLocation": {"const": "https://json-schema.org/tests/content/draft-next/type/1"},
23+
"instanceLocation": {"const": ""},
24+
"annotations": false,
25+
"errors": {
26+
"properties": {
27+
"type": {"type": "string"}
28+
},
29+
"required": ["type"]
30+
}
31+
},
32+
"required": ["evaluationPath", "schemaLocation", "instanceLocation"]
33+
}
34+
}
35+
},
36+
"required": ["details"]
37+
}
38+
}
39+
}
40+
]
41+
}
42+
]

‎output-tests/content/draft2019-09/general.json

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
[
2+
{
3+
"description": "failed validation produces no annotations",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/2020-12/schema",
6+
"$id": "https://json-schema.org/tests/content/draft2019-09/general/1",
7+
"type": "string",
8+
"readOnly": true
9+
},
10+
"tests": [
11+
{
12+
"description": "readOnly annotation is dropped",
13+
"data": 1,
14+
"output": {
15+
"basic": {
16+
"type": "object",
17+
"properties": {
18+
"errors": {
19+
"type": "array",
20+
"items": {
21+
"properties": {
22+
"annotation": false
23+
}
24+
}
25+
},
26+
"annotations": false
27+
},
28+
"required": ["errors"]
29+
}
30+
}
31+
}
32+
]
33+
}
34+
]

‎output-tests/content/draft2019-09/readOnly.json

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
[
2+
{
3+
"description": "readOnly generates its value as an annotation",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/2019-09/schema",
6+
"$id": "https://json-schema.org/tests/content/draft2019-09/readOnly/1",
7+
"readOnly": true
8+
},
9+
"tests": [
10+
{
11+
"description": "readOnly is true",
12+
"data": 1,
13+
"output": {
14+
"basic": {
15+
"type": "object",
16+
"properties": {
17+
"annotations": {
18+
"type": "array",
19+
"contains": {
20+
"type": "object",
21+
"properties": {
22+
"keywordLocation": {"const": "/readOnly"},
23+
"absoluteKeywordLocation": {"const": "https://json-schema.org/tests/content/draft2019-09/readOnly/1#/readOnly"},
24+
"instanceLocation": {"const": ""},
25+
"annotation": {"const": true}
26+
},
27+
"required": ["keywordLocation", "absoluteKeywordLocation", "instanceLocation", "annotation"]
28+
}
29+
},
30+
"errors": false
31+
},
32+
"required": ["annotations"]
33+
}
34+
}
35+
}
36+
]
37+
}
38+
]

‎output-tests/content/draft2019-09/type.json

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
[
2+
{
3+
"description": "validating type",
4+
"schema": {
5+
"$schema": "https://json-schema.org/draft/2019-09/schema",
6+
"$id": "https://json-schema.org/tests/content/draft2019-09/type/1",
7+
"type": "string"
8+
},
9+
"tests": [
10+
{
11+
"description": "incorrect type must be reported, but a message is not required",
12+
"data": 1,
13+
"output": {
14+
"basic": {
15+
"type": "object",
16+
"properties": {
17+
"errors": {
18+
"contains": {
19+
"type": "object",
20+
"properties": {
21+
"keywordLocation": {"const": "/type"},
22+
"absoluteKeywordLocation": {"const": "https://json-schema.org/tests/content/draft2019-09/type/1#/type"},
23+
"instanceLocation": {"const": ""},
24+
"annotation": false,
25+
"error": {"type": "string"}
26+
},
27+
"required": ["keywordLocation", "absoluteKeywordLocation", "instanceLocation"]
28+
}
29+
}
30+
},
31+
"required": ["errors"]
32+
}
33+
}
34+
},
35+
{
36+
"description": "correct type yields an output unit",
37+
"data": 1,
38+
"output": {
39+
"basic": {
40+
"type": "object",
41+
"properties": {
42+
"annotations": {
43+
"contains": {
44+
"type": "object",
45+
"properties": {
46+
"valid": {"const": true},
47+
"keywordLocation": {"const": "/type"},
48+
"absoluteKeywordLocation": {"const": "https://json-schema.org/tests/content/draft2019-09/type/1#/type"},
49+
"instanceLocation": {"const": ""},
50+
"annotation": false,
51+
"error": false
52+
},
53+
"required": ["keywordLocation", "absoluteKeywordLocation", "instanceLocation"]
54+
}
55+
}
56+
},
57+
"required": ["annotations"]
58+
}
59+
}
60+
}
61+
]
62+
}
63+
]

0 commit comments

Comments
 (0)