Looking into vendoring the JSON Schema for config (for check-jsonschema)

[sirosen]

👋 Hi there, check-jsonschema maintainer here!

I have a recent user contribution to try to add the JSON Schema to check-jsonschema: python-jsonschema/check-jsonschema#324

Unfortunately, there are some issues with this, and as I've done some reading and looking into the mkdocs-material schema, I'm finding just how complex the situation is .
check-jsonschema targets offline usage in pre-commit.ci as one of its use-cases, which means that relying on lazy $ref resolution isn't viable. Schemas are vendored with the tool and need to be collected as part of the package, not fetched at runtime. I could compromise on this, but I'm pretty concerned with how confusing that would be for users -- I'd rather just not add a relevant hook and let users manually specify --schemafile "https://squidfunk.github.io/mkdocs-material/schema.json" if they want to use our tools together.

So I'm now thinking about what it would take to vendor the whole universe of schemas which are referenced from that starting point (docs/schema.json). It seems like rather a lot of work to even try.

Most of the technical challenges, the "can it be done?" bits are of interest to me, though some are hard enough to give me some pause. I have concerns primarily about "should it be done?" and making some decisions about handling 3rd party inclusions.
I'd appreciate any feedback from the maintainers of mkdocs-material on this front in particular.

• First off, does this sound like a bad idea? Is there anything which makes you worry about the idea of someone out there trying to pack up the family of schemas for reuse offline?
• Other than plugins.json, are there other points which reference schemas outside of the repo?
• Suppose that at vendoring-time, not all schemas can be retrieved (errors). How should I resolve this? Any thoughts or ideas?
  • If we can't get things from squidfunk.github.io we can consider this a hard-error.
  • If we can't retrieve a new external schema (no past vendored copy), just ignore it? (fails at runtime, maybe fine?)
  • If we can't retrieve an existing external schema (there is a past vendored copy), I think we just keep the old one. Tell me if that seems risky. (What are the alternatives?)
• If vendoring completely excluded 3rd party schemas, do we have any rough sense or guess about how many users (by percentage) that would include or exclude?
  • I think this would allow me to scrape schemas purely from the docs/ dir of the repo; does that sound right?
• Vendoring is only partly automated today. I get an automatic PR which updates daily, and manually merge and release newly vendored data periodically. For most of schemastore this is fine and is usually only a couple of weeks out of date. Would the mkdocs-material schema suffer greatly by being a bit out of date? (Beyond the obvious "missing bugfixes" concerns, etc.)

---

I'll note that this is mostly academic for right now. I've convinced myself that this isn't a good "unwinding after work" project I can knock out in a couple of hours and requires a lot more thought.
But I'm curious if there's an immediate 👍 or 👎 reaction from the maintainers, and if there are particular thoughts about how to go about tackling this if I want to give it a try.

[squidfunk]

Thanks for considering including our schema!

So I'm now thinking about what it would take to vendor the whole universe of schemas which are referenced from that starting point (docs/schema.json). It seems like rather a lot of work to even try.

I'm not sure what you mean by vendoring. Inline all external schemas into one? That's not feasible for us, but you could certainly post-process the schema and integrate them all into one. There are many third party plugins that integrate with Material for MkDocs for which we just don't have the time to maintain them on our side, so we're happy to include and link to them. If you search for mkdocs plugin on GitHub, you get 350 results, and there are probably plugins that are not on GitHub. Far too many.

Other than plugins.json, are there other points which reference schemas outside of the repo?

AFAIK, no.

Suppose that at vendoring-time, not all schemas can be retrieved (errors). How should I resolve this? Any thoughts or ideas?

Probably just ignore them, but YMMV.

If vendoring completely excluded 3rd party schemas, do we have any rough sense or guess about how many users (by percentage) that would include or exclude?

I'm, sorry, I don't have numbers on schema use.

Vendoring is only partly automated today. I get an automatic PR which updates daily, and manually merge and release newly vendored data periodically. For most of schemastore this is fine and is usually only a couple of weeks out of date. Would the mkdocs-material schema suffer greatly by being a bit out of date? (Beyond the obvious "missing bugfixes" concerns, etc.)

Probably not, but again, YMMV.

[alexvoss]

This explains vendoring. IMHO, to include the schema.json from mkdocs-material here would require additional work since it has been written as a helper for people editing, not as a way to validate mkdocs.yml files. Adding a random plugin that is not already explicitly supported would throw validation errors, which is ok in an editor but not in a pre-commit hook. Not sure we would do ourselves much of a favor by 'vendoring' the schema included in mkdocs-material.

There is another mkdocs.yml schema out there but I do not know where it comes from, who maintains it. It does not seem to be in the MkDocs repo, where I think it should live. If there was interest in doing this, I would happily collaborate on updating this if needed. It should also be hosted in a way that its origin is traceable.

Apologies if I missed something/am holding the wrong stick. Am fairly new to JSON schemas.

[sirosen]

I'm not sure what you mean by vendoring. Inline all external schemas into one?

Not necessarily inlining, just pulling them all in-tree into check-jsonschema.
My goal would be to pull in a copy of all schemas referenced starting from schema.json, and then arrange them in the right structure in memory when doing validation so that no external network calls are needed.

If mkdocs-material were willing to provide a fully inlined version of the schema (except for external plugins), I would use that very happily.
I'd be more than glad to contribute a tool either here or on pypi (name TBD, jsonschema-inliner?) which inlines everything into schema-inlined.json -- I'm happy to be the one who does the legwork, but I care about the SOC and keeping check-jsonschema isolated from that process.
But I don't want to write such a thing if mkdocs-material isn't interested in using it.

If you search for mkdocs plugin on GitHub, you get 350 results, and there are probably plugins that are not on GitHub. Far too many.
...
I'm, sorry, I don't have numbers on schema use.

Thanks; this gets me a better handle on the scope of what I'm looking into! It's the sort of info I thought might be more understandable to the maintainers here than to me, as an outsider.

And no trouble about not having stats or numbers. I figured it was a long shot to ask if we have any sense of usage trends (hard to track).

Suppose that at vendoring-time, not all schemas can be retrieved (errors). How should I resolve this? Any thoughts or ideas?

Probably just ignore them, but YMMV.

Just to share, validation under jsonschema can't really ignore a schema, or I guess more precisely it demands that we define "what it means to ignore it".

I could stub them out with true, which would be a lot like ignoring a missing subschema.
But to me "always passing" vs "always failing" vs "always passing but triggering a warning" is the kind of decision I would need to make.
Or there's always a path like "if present in anyOf/oneOf, remove it from the list".

Maybe I'm being too pedantic, but these subtleties define the failure modes of the schema.
e.g., If you have anyOf: [{...}, {...}, true], then that always passes, even if you have relevant subschemas but true is in that list. So... it's messy.

IMHO, to include the schema.json from mkdocs-material here would require additional work since it has been written as a helper for people editing, not as a way to validate mkdocs.yml files. Adding a random plugin that is not already explicitly supported would throw validation errors, which is ok in an editor but not in a pre-commit hook.

This is making me think the payoff from this project is not worth the effort it would cost.
If unrecognized plugins would always fail validation, it's likely that we'd always be working at cross purposes.

It at least gives me pause.

There is another mkdocs.yml schema out there but I do not know where it comes from, who maintains it. It does not seem to be in the MkDocs repo, where I think it should live. If there was interest in doing this, I would happily collaborate on updating this if needed. It should also be hosted in a way that its origin is traceable.

Given that schemastore is completely community driven on github, you can trace their schemata by cracking open their repo and looking at the history. In this case: SchemaStore/schemastore#2768
That "community driven" ethos is a double-edged sword. IMO, each schema in schemastore should be looked at as its own independent project, with its own maintenance history.

I think that's a generic "mkdocs" schema though, not specific to mkdocs-material. It may be possible that I'm the one holding the wrong stick now though. 😉

---

Thanks all for the feedback. I have to keep thinking about this, but I'm starting to tend away from trying to pursue this feature-set. It just seems like "a lot" for something of questionable value.