Config Structural Validation using a declarative JSON Schema Part 0: Scripts, Global

[Zash]

Warning

This is public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request
• business confidential information, such as customer names

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• kind/adr

Release notes

The configuration files are now validated against a JSON Schema.

Platform Administrator notice

When validating config against the schema you may encounter cases where they disagree on what type a field should be and it may be necessary to tweak either the config or the schema to match the intent.

What does this PR do / why do we need this PR?

...

• Fixes [5]Add structural config validation #1427

Additional information to reviewers

This adds a huge JSON Schema file (in YAML format) that describes the structure of the config. Feel free to ignore it when reviewing, it does not need to be perfect immediately but can be incrementally perfected over time.

A new tool, yajsv, written in Go, is added as requirement which needs to be installed (bin/install-requirements.bash). Since JSON Schema is an an open specification with multiple implementations, it could be replaced in the future. We could even write our own using one of the available libraries.

To try this, run e.g.:

./bin/ck8s validate sc

Information about JSON schema can be found at https://json-schema.org/

The specifications themselves are found at

• https://json-schema.org/draft/2020-12/json-schema-core (basics)
• https://json-schema.org/draft/2020-12/json-schema-validation (definitions of many kinds of constraints)

Notes about certain common schema fields:

• default can be used to describe a default value, which could be used in future tooling to actually fill in default values, but for now serves as documentation.
• example and examples can contain one or many example values. This could be shown in future tooling to improve UX.
• format allows specifying what kind of string to expect. Common usable values include hostname, uri (for URLs etc), email. Unknown values are ignored by the validator, so common formats could be described and potentially implemented in future tooling.
• additionalProperties defines schema for a property of an object that is not covered by properties, setting this to false rejects unknown values, and thus detects things that might have been missed.

Screenshots

$ ./bin/ck8s validate sc
[ck8s] Validating sc config
[ck8s] Failed schema validation:
config-sc.yaml: nodeLocalDns.customConfig: Invalid type. Expected: string, given: null
config-sc.yaml: falco.driver.ebpf.path: Invalid type. Expected: string, given: null
config-sc.yaml: certmanager.tolerations: Invalid type. Expected: array, given: object
config-sc.yaml: certmanager.cainjector.tolerations: Invalid type. Expected: array, given: object
config-sc.yaml: certmanager.webhook.tolerations: Invalid type. Expected: array, given: object
config-sc.yaml: certmanager: Must validate all the schemas (allOf)
config-sc.yaml: networkPolicies.global.externalLoadBalancer: Invalid type. Expected: string, given: boolean
config-sc.yaml: networkPolicies.global.ingressUsingHostNetwork: Invalid type. Expected: string, given: boolean
1 of 1 failed validation
[ck8s] Do you want to abort? (y/n): 

Can provide validation and help messages in e.g. VS Code:

Could also be used to generate documentation:

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • The logs do not show any errors after the change
• Network Policy checks:
  • Any changed pod is covered by Network Policies
  • The change does not cause any dropped packages in the NetworkPolicy Dashboard
• Pod Security Policy checks:
  • Any changed pod is covered by Pod Security Admission
  • Any changed pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Bug checks:
  • The bug fix is covered by regression tests

[robinAwallace]

As the schema very long I have not really gone though it. But it looks promising 🙂 I will try to try it out next week.

[robinAwallace]

I have manged to try it out now and it is very nice to get error on specific values when they are "wrong" 👍
But I have some thoughts. 🙂

[Zash]

In the pipeline, where does this falco setting come from?

[ck8s] Failed schema validation:
wc-config.yaml: falco.alerts.enabled: Invalid type. Expected: boolean, given: string
1 of 1 failed validation

[aarnq]

Please add yajsv into check_tools and into the tests/Dockerfile.

[aarnq]

I've created a PR with a helper to regenerate the full diff that is used for the update-ips tests.

[Zash]

thoughts on split vs single schema file

split schema into multiple files

pro:

• easier review with smaller files
• easy to validate individual sections

con:

• some tools don't support external references, e.g. that documentation generation tool
• some tools have poor handling of multi-file schema, e.g. doesn't report which file to look in or reports a 404 error
• breaks the symmetry-ish between config and schema
• reference by URL seems needlessly complicated
• tools may try to download resources over the internet

single schema

pro:

• simpler references, just $ref: "#/path/somewhere" instead of a full URL and needing $id
• a larger schema may work as a pressure towards smaller schema, and thus smaller config with more reuse

con:

• harder to review
• harder to navigate?

conclusion

tools vs tools tradeoff? I prefer single file. can always switch later.

[aarnq]

thoughts on split vs single schema file

Yeah, the single schema sounds better in this case.

pro:

• a larger schema may work as a pressure towards smaller schema, and thus smaller config with more reuse

And this I especially like 😁

I assume the idea is to still have a separate one for secrets, as we don't merge that in?

[robinAwallace]

thoughts on split vs single schema file

tools vs tools tradeoff? I prefer single file. can always switch later.

Very nice summary of the pros and cons. 👍 I agree stick with a single file.

[Zash]

I assume the idea is to still have a separate one for secrets, as we don't merge that in?

Yeah, assuming secrets is its own thing. Different kind of thing having its own schema seems sensible. And given the sensitive nature, keeping them separate as long as possible to minimize exposure is probably good.

[Zash]

✅ All checks have passed

🎉

Now to add more tests specifically for schema...

[Zash]

Gonna rebase, reorder and squash a bit.

[robinAwallace]

LGTM 🥳 Im happy to merge this. Just one thought, could you mention some were how to setup vscode to work with the validator? 🙂

[Zash]

could you mention some were how to setup vscode to work with the validator? 🙂

As in, enabling it here or enabling it somewhere else where you have config?

I can write some extra words about it in .../schema/README.md, it should mainly be having an already recommended plugin and some references in .vscode/settings.json

[robinAwallace]

I can write some extra words about it in .../schema/README.md, it should mainly be having an already recommended plugin and some references in .vscode/settings.json

Yeah, I think it is worth mentioning that, that plugin would enable validation/help in vscode. As you basically already done in this PR description.

[Zash]

The remaining parts of config/schemas/config.yaml are now in

• Config Structural Validation Schema Part 1: Commons, HNC, CAPI, Harbor, Storage #2063
• Config Structural Validation Schema Part 2: Security and Policy #2064
• Config Structural Validation Schema Part 3: Networking, Ingress, DNS #2065
• Config Structural Validation Schema Part 4: Monitoring #2066

Each one adds a few more or less related sections to the config schema.

Five is a good number.