feat(docs): Formal Schemas for Signed Docs (#208)

* docs(docs): Add wip files for formal signed document schemas

* docs: wip

* feat: wip

* feat: Signed Docs formal specs WIP

* feat: Add draft7 jsonschema to meta templates schema definitions

* feat: Add CI actions to check generated data matches schema

* fix: try and fix project name for CI

* fix: remove obsolete files

* fix: fix markdown

* fix: spelling

* fix: Make sure json version of document specifications is sorted to make change tracking easier.

* feat: Add base types information to the json version of the document specifications

* fix: add recommended justfile extension for vscode

* feat: Example of building docs directly from specification data

* fix: spelling

* fix: Spelling and Markdown

* feat: Add documentation auto generation for new signed documents

* feat: Add status which allows a reference to define one or multiple references.

* feat: Add three states to document submission and change the names to more align with intent vs business logic behaviour.

* feat: Rationalize ref/ref_hash and collation into a single `ref` metadata field.

* feat: Fix validation issues

* feat: Signed Doc defintion improvements WIP

* feat: Signed docs Docs generation WIP

* feat(docs): Formal signed doc specs auto markdown generation WIP

* feat(docs): Add start of individual documentation pages for each signed document defined

* feat: individual document specifications now generated

* feat: Add generation of document realtionship diagram

* fix: Relationship diagram generation

* fix: Build signed doc specs into the architecture documentation

* fix: remove unused link

* fix: spelling

* fix: spec index needs sub documents

* fix: Put all metadata in doc diafram.  Add doc diagram to pages

* fix: Remove single doc diagram untilwe fix links

* fix: Add individual document relationship images to each page

Author: stevenj

.config/dictionaries/project.dic

@@ -51,6 +51,7 @@ crontabs
 crontagged
 csprng
 cstring
+cuelang
 dalek
 dashmap
 Datelike
@@ -123,6 +124,7 @@ jorm
 jormungandr
 Jörmungandr
 jsonschema
+Justfile
 kiduri
 lcov
 Leay
@@ -141,6 +143,7 @@ logcall
 lookaside
 maindbname
 mapref
+markdownlint
 mdlint
 mdns
 MEMMAP
@@ -200,6 +203,7 @@ pubkey
 publickey
 pubspec
 pwrite
+pytest
 qpsg
 quic
 rapidoc

.gitignore

@@ -84,3 +84,7 @@ $RECYCLE.BIN/
 # Dart stuff
 /pubspec.lock
 /.dart_tool/**/*
+
+# Python stuff
+.pytest_cache
+__pycache__

.markdownlint-cli2.jsonc

@@ -10,7 +10,8 @@
   "ignores": [
     ".config/dictionaries/**",
     "**/target/**",
-    "**/.dart_tool/**"
+    "**/.dart_tool/**",
+    "**/.pytest_cache/**"
   ],
   // Set standard config options in `/.markdownlint.jsonc`
   "config": {

.vscode/extensions.json

@@ -24,5 +24,8 @@
         "dtsvet.vscode-wasm",
         "terrastruct.d2",
         "fill-labs.dependi",
+        "nefrob.vscode-just-syntax",
+        "charliermarsh.ruff",
+        "ms-python.python",
     ]
 }

Justfile

@@ -15,6 +15,19 @@ check-spelling:
     earthly +check-spelling
 
 
+# Fix and Check Markdown files
+format-python-code:
+    ruff check --select I --fix .
+    ruff format .
+
+# Fix and Check Markdown files
+lint-python:
+    ruff check .
+
+# generates specifications data
+gen_specs:
+    just specs/pre-push
+
 # Pre Push Checks - intended to be run by a git pre-push hook.
-pre-push: check-markdown check-spelling
+pre-push: gen_specs check-markdown check-spelling format-python-code lint-python
     just rust/pre-push

docs/Earthfile

@@ -4,6 +4,7 @@ IMPORT github.com/input-output-hk/catalyst-ci/earthly/docs:v3.3.0 AS docs-ci
 
 
 IMPORT .. AS repo
+IMPORT ../specs AS specs
 
 # Copy all the source we need to build the docs
 src:
@@ -13,6 +14,9 @@ src:
     # Now copy into that any artifacts we pull from the builds.
     COPY --dir repo+repo-docs/repo /docs/includes
 
+    # Copy our generated Signed Document Specification data.
+    COPY specs+src/signed_doc.json /docs/includes
+
 
 # Build the docs here.
 docs:

docs/src/architecture/08_concepts/catalyst_docs/.pages

@@ -1,5 +1 @@
 title: Catalyst Documents
-arrange:
-- proposal.md
-- review.md
-- comment.md

docs/src/architecture/08_concepts/catalyst_docs/comment.md

@@ -46,17 +46,17 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   "content-encoding" => "br"
   ```
 
-* [`ref`](./../signed_doc/meta.md#ref-document-reference).
+* [`ref`](./../signed_doc/metadata.md#ref-document-reference).
   Reference to a related [Proposal Document],
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   [proposal document `type`][Proposal Document] field value.
 
-* [`template`](./../signed_doc/meta.md#ref-document-reference).
+* [`template`](./../signed_doc/metadata.md#ref-document-reference).
   A reference to the comment template document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   [comment template `type`](#comment-template) field value.
 
-* [`reply`](./../signed_doc/meta.md#reply-reply-reference) (optional).
+* [`reply`](./../signed_doc/metadata.md#reply-reply-reference) (optional).
   A reference to another comment document,
   where the comment is in reply to the referenced comment.
   The [`type`](./../signed_doc/spec.md#type) of the replied document
@@ -65,7 +65,7 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   The referenced `comment` must be for the same proposal [`id`](./../signed_doc/spec.md#id),
   but can be for a different proposal [`ver`](./../signed_doc/spec.md#ver).
 
-* [`section`](./../signed_doc/meta.md#section-section-reference) (optional).
+* [`section`](./../signed_doc/metadata.md#section-section-reference) (optional).
   Used when the comment only applies to a specific section to the document being commented upon,
   and not the entire document.
 
@@ -106,7 +106,7 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   "content-encoding" => "br"
   ```
 
-* [`category_id`](./../signed_doc/meta.md#category_id) (optional).
+* [`category_id`](./../signed_doc/metadata.md#category_id) (optional).
   A reference to the category document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   `48c20109-362a-4d32-9bba-e0a9cf8b45be` value.

docs/src/architecture/08_concepts/catalyst_docs/proposal.md

@@ -50,12 +50,12 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   "content-encoding" => "br"
   ```
 
-* [`template`](./../signed_doc/meta.md#ref-document-reference).
+* [`template`](./../signed_doc/metadata.md#ref-document-reference).
   A reference to the proposal template document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   [proposal template `type`](#proposal-template) field value.
 
-* [`category_id`](./../signed_doc/meta.md#category_id) (optional).
+* [`category_id`](./../signed_doc/metadata.md#category_id) (optional).
   A reference to the category document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   `48c20109-362a-4d32-9bba-e0a9cf8b45be` value.
@@ -112,7 +112,7 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   "content-encoding" => "br"
   ```
 
-* [`category_id`](./../signed_doc/meta.md#category_id) (optional).
+* [`category_id`](./../signed_doc/metadata.md#category_id) (optional).
   A reference to the category document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   `48c20109-362a-4d32-9bba-e0a9cf8b45be` value.

docs/src/architecture/08_concepts/catalyst_docs/review.md

@@ -46,7 +46,7 @@ A list of used [Catalyst Signed Document protected header fields](./../signed_do
   "content-encoding" => "br"
   ```
 
-* [`template`](./../signed_doc/meta.md#ref-document-reference).
+* [`template`](./../signed_doc/metadata.md#ref-document-reference).
   A reference to the review template document,
   which [`type`](./../signed_doc/spec.md#type) must be equal to
   [review template `type`](#review-template) field value.