Separate schema registry and TypeScript code generation responsibilities, add docs (#807)

* Separate schema registry and Typescript type generation responsibilities

Improve inline documentation

* Create new explanation dir in contributor guide

Move architecture documentation in

* Re-organize contributor how-tos into how-tos subdir

* Create interface directory before attempting to write to it

* Fix broken link

* Update toctree glob expressions to include files in same dir

* Fix link

* More consistent how-to titles

* Add code generation documentation

* Add more specificity about code generation commands

* Lint 🔔

* Fix doc build "misc.highlighting_failure" failure

* Adjust extension order to fix rendering

If the extension is loaded after sphinx_tabs, it doesn't work! :o

* Switch flowchart to top-down

It was rendering too small in left-to-right mode

* Fix mermaid rendering issues

Some lines are too long. Handle special characters with quotes.

Author: mfisher87

docs/conf.py

@@ -16,6 +16,7 @@
     "sphinx_autodoc_typehints",
     "sphinx.ext.intersphinx",
     "sphinx.ext.napoleon",
+    "sphinxcontrib.mermaid",
     "sphinx_tabs.tabs",
     "sphinx_exercise",
     "sphinx_togglebutton",
@@ -25,6 +26,7 @@
 myst_enable_extensions = [
     "colon_fence",
 ]
+myst_fence_as_directive = ["mermaid"]
 
 master_doc = "index"
 project = "JupyterGIS"

docs/contributor_guide/architecture.md renamed to docs/contributor_guide/explanation/architecture.md

@@ -0,0 +1,105 @@
+# Code generation
+
+JupyterGIS leverages code generation to share information about data structures across
+Python and TypeScript packages.
+
+## Overview
+
+There are 3 code generation targets: the schema registry, TypeScript types, and Python
+types.
+
+Code generation tasks are defined in `packages/schema/package.json`.
+You can run all code generation tasks with `jlpm run build:schema`, or run them
+individually using the commands in the diagram below.
+
+```mermaid
+flowchart TD
+    package-manifest["package.json"]
+    npm-script-schema-registry{{"build:schema:registry"}}
+    npm-script-schema-js{{"build:schema:js"}}
+    npm-script-schema-py{{"build:schema:py"}}
+
+    combined-schema[("Combined schema<br/>(forms.json)")]
+    schema-registry[["schemaregistry.ts"]]
+    react-forms[["React forms"]]
+
+    ts-types[("TypeScript type definitions")]
+
+    tmp-schema[("Temporary schema<br/>(rewritten $ref paths)")]
+    py-types[("Python type definitions")]
+
+    package-manifest -->|jlpm run| npm-script-schema-registry
+    package-manifest -->|jlpm run| npm-script-schema-js
+    package-manifest -->|jlpm run| npm-script-schema-py
+
+    npm-script-schema-registry -->|scripts/dereference-and-combine-schemas-for-registry.js| combined-schema
+    -->|read| schema-registry
+    -->|react-jsonschema-form| react-forms
+
+    npm-script-schema-js -->|json-schema-to-typescript| ts-types
+    npm-script-schema-js -->|scripts/add-schema-version.ts| ts-types
+
+    npm-script-schema-py -->|scripts/preprocess-schemas-for-python-type-generation.js| tmp-schema
+    --> |datamodel-code-generator| py-types
+```
+
+## Schema registry
+
+The schema registry is built for use in TypeScript code.
+It's used by [`react-jsonschema-form` (RJSF)](https://github.com/rjsf-team/react-jsonschema-form)
+to generate forms (React components) from JSONSchema.
+
+Prior to generating the schema registry, we dereference (i.e. inline `$ref`s) our
+schemas and combine them into one JSON file which contains a mapping from schema names
+to schema data for each schema.
+
+:::{note}
+Combining the schemas into `forms.json` is vestigial.
+Before we had a schema registry, we would directly index into this data to find a
+schema.
+Now that we're building a schema registry for interfacing with schemas, perhaps we don't
+need a combined `forms.json` schema.
+:::
+
+## TypeScript types
+
+TypeScript types are generated from the JSONSchema files using
+[`json-schema-to-typescript`](https://github.com/bcherny/json-schema-to-typescript).
+
+We additionally run a custom script (`scripts/add-schema-version.ts`) to generate a
+version number variable.
+
+## Python types
+
+Python types are generated from the JSONSchema files using
+[`datamodel-code-generator`](https://github.com/koxudaxi/datamodel-code-generator).
+
+### Weirdness with `$ref` paths
+
+Unfortunately, `datamodel-code-generator` expects `$ref` paths to be expressed
+differently from `json-schema-to-typescript`.
+The former expects "relative" paths, and the latter expects "absolute" paths.
+
+For example, if we have two schema files:
+
+```text
+├── referent.json
+└── subdir
+    └── referrer.json
+```
+
+...`json-schema-to-typescript` would expect `referrer.json` to contain a reference like:
+
+```json
+"$ref": "referent.json",
+```
+
+...and `datamodel-code-generator` would expect the reference in `referrer.json` to look like:
+
+```json
+"$ref": "../referent.json",
+```
+
+**For this reason, we chose to write our schema files in the way
+`json-schema-to-typescript` expects, and pre-process them into a temp directory so they
+look the way `datamodel-code-generator` expects before generating Python types.**

docs/contributor_guide/explanation/code-generation.md

@@ -0,0 +1,8 @@
+# Explanation
+
+```{toctree}
+:maxdepth: 1
+:glob:
+
+*
+```

docs/contributor_guide/explanation/index.md

@@ -1,4 +1,4 @@
-# Building JupyterGIS documentation locally
+# Build JupyterGIS documentation locally
 
 :::{tip}
 You can use `conda` or `mamba` as drop-in replacements for `micromamba` in the steps
@@ -7,7 +7,7 @@ below, but they will not be as fast.
 
 ## 0. Build JupyterGIS JavaScript packages
 
-Follow the [development environment setup instructions](./development_setup.md) through running `jlpm run build` from the **root of the repo**.
+Follow the [development environment setup instructions](../development_setup.md) through running `jlpm run build` from the **root of the repo**.
 
 :::{important}
 Navigate to the `docs/` directory before starting any of the following steps!

docs/contributor_guide/docs.md renamed to docs/contributor_guide/how-tos/build-docs-locally.md

@@ -1,7 +1,7 @@
-# Code quality
+# Check code quality
 
 ```{seealso}
-Complete [dev install](./development_setup) instructions before continuing.
+Complete [dev install](../development_setup) instructions before continuing.
 ```
 
 We have several tools configured for checking code quality:

docs/contributor_guide/code_quality.md renamed to docs/contributor_guide/how-tos/code_quality.md

@@ -5,4 +5,5 @@
 :glob:
 
 */*
+*
 ```

docs/contributor_guide/how-tos/index.md

@@ -1,4 +1,4 @@
-# Releasing
+# Release JupyterGIS packages
 
 ## Automated Releases with `jupyter_releaser`
 

docs/contributor_guide/releasing.md renamed to docs/contributor_guide/how-tos/release.md

@@ -1,4 +1,4 @@
-# Testing
+# Test JupyterGIS
 
 ## Python unit tests
 

docs/contributor_guide/testing.md renamed to docs/contributor_guide/how-tos/test.md

@@ -1,4 +1,4 @@
-# Tutorial Writing Guide
+# Write tutorials
 
 This guide provides best practices for writing tutorials for JupyterGIS. To keep tutorials consistent and easy to follow, please follow the guidelines below.