Schema test coverage: which thresholds are useful?

[ralfhandl]

The new JSON Schema Test Coverage tool we are now using allows to set separate coverage thresholds for

• Lines
• Statements = JSON Schema keywords
• Functions = subschemas
• Branches = true/false branches for each keyword (except for keywords that don't branch such as annotation-only keywords like title and description)

Suggestion

Set coverage thresholds of

• 100% for lines
• 100% for keywords (statements)
• 100% for subschemas (functions)
• no threshold for branches

Rationale

Lines and Statements

The natural threshold for lines and statements is 100% - if we have a line/keyword in our schema that we don't touch in at least one test case, then someone could remove that line/keyword and the automated PR status check for schema tests would still be green.

It also helps spotting test cases that fail for the wrong reason, such as the two test cases that are now being fixed by

• 3.2 fix test cases: reusable request bodies need a name #4840

These were intended to cover schema lines

OpenAPI-Specification/src/schemas/validation/schema.yaml

Lines 604 to 608 in 805084d

	- dependentSchemas:
	encoding:
	properties:
	prefixEncoding: false
	itemEncoding: false

and failed before reaching that part of the schema due to another unintended failure condition.

Subschemas (Functions)

Subschema coverage includes

• the keyword unevaluatedProperties which takes a subschema as its argument, and
• the keyword properties which takes a map as its argument where the map values are subschemas.

This means that we can only be sure to have covered all properties in our schema if we strive for subschema (function) coverage of 100%.

Branches

Covering 100% of all branches seems less useful because for a keyword use such as

OpenAPI-Specification/src/schemas/validation/schema.yaml

Lines 844 to 846 in 805084d

	type:
	- object
	- boolean

the corresponding two branches can be covered with a "pass" test case providing an object, and a "fail" test case providing an array, which leaves both good cases (providing a boolean) or bad cases (providing a number, a string, or null) uncovered.

[ralfhandl]

Pull request

• 3.2 schema tests: 100% subschema (function) coverage #4857

shows how to cover unevaluatedProperties: true with a single "fail" test case, avoiding clutter in the test case folders.