[chore] doc refactoring

Author: cscheid

README.md

@@ -64,5 +64,4 @@ Parses [quarto-web](https://github.com/quarto-dev/quarto-web) with a small numbe
 
 - actually good error messages
 - fine-grained source location tracking
-- WASM distribution
 - solve glaring performance issues

crates/tree-sitter-qmd/README.md

@@ -5,31 +5,8 @@ It has a number of non-standard syntax that are used by Quarto in .qmd files spe
 
 The original `tree-sitter-markdown` grammar was written by [Matthias Deiml](https://github.com/MDeiml).
 
-At a high level, tree-sitter-qmd should be used like tree-sitter-md, by using the results of a block parse
-to trigger the inline grammar parsing.
+`tree-sitter-qmd` is an internal package written entirely to support `quarto-markdown-pandoc`.
 
 For the original tree-sitter-md readme, see [README.tree-sitter-md.md].
 
-## QMD changes/additions
-
-- A fixed set of extensions
-
-- shortcode syntax
-
-- attribute syntax
-
-  - including Quarto's `{lang}` syntax that isn't commonmark
-
-  - including raw block/inline `{=format}` syntax
-
-## QMD _REMOVALS_
-
-- Simplified link syntax. the only link syntax supported are _inline links_: `[text](destination title)`
-
-  - no wikilink support
-
-  - no shortcut reference link support, we use that syntax for spans instead
-
-- Similarly, the only image syntax supported is the one corresponding to inline links: `![text](image-name title)`
-
-- no HTML support: QMD is meant to translate into more than one syntax. We use rawblock instead.
+For more information on the syntax supported by quarto-markdown, see the top-level docs folder, and specifically the [syntax-nodes.md file](../../docs/syntax-notes.md).

docs/syntax-notes.md

@@ -8,17 +8,31 @@ We aim to be largely compatible with Pandoc's `markdown` and `Commonmark` format
 
 Syntax extensions are handled by [desugaring](https://cs.brown.edu/courses/cs173/2012/book/Desugaring_as_a_Language_Feature.html) into regular Pandoc AST nodes.
 
+### Scoped metadata
+
+Our intermediate representation can store a metadata block inside the document, allowing (in principle)
+for metadata in the document to be scoped to a particular portion of the document.
+
 ### Shortcodes
 
 We have "native" shortcode support in the "Pandoc" AST in pandoc.rs, and
 we desugar them to Pandoc spans in a Rust filter.
 
-### Notes
+### Footnotes
 
 We parse footnotes differently from Pandoc.
 We use NoteReference (Inline) and NoteDefinition (block) nodes.
 These are desugared into spans and divs in a Rust filter.
 
+### Editor markup
+
+Inspired by [CriticMarkup](https://fletcher.github.io/MultiMarkdown-6/syntax/critic.html) and [djot](https://djot.net), Quarto offers syntax for edit marks:
+
+- Insertions: `[++ Insert this markdown content]`
+- Deletions: `[-- Delete this sentence]`
+- Highlighting: `[!! this will be highlighted in rendering]`
+- Comment: `[>> this will show up as a comment]`
+
 ### Reader raw blocks
 
 Quarto Markdown supports the following syntax:
@@ -200,4 +214,16 @@ We will also not support definition lists directly.
 Consider `^[footnote-or-span]{.class}^`. `^[` denotes both the start of a footnote and potentially the combination of a superscript block with a span; this parse is ambiguous.
 
 Quarto-markdown's parser prefers the footnote interpretation. In case an immediately nested span is needed, use a space between `^` and `[`.
-Superscript nodes with leading spaces are disallowed in Pandoc, but Quarto-markdown will trim spaces.
+Superscript nodes with leading spaces are disallowed in Pandoc, but Quarto-markdown will trim spaces.
+
+## QMD _REMOVALS_
+
+- Simplified link syntax. the only link syntax supported are _inline links_: `[text](destination title)`
+
+  - no wikilink support
+
+  - no shortcut reference link support, we use that syntax for spans instead
+
+- Similarly, the only image syntax supported is the one corresponding to inline links: `![text](image-name title)`
+
+- no HTML support: QMD is meant to translate into more than one syntax. We use rawblock instead.

docs/testing.md

@@ -0,0 +1,15 @@
+## Testing
+
+tree-sitter-markdown-inline has a tree-sitter test suite that can be run with
+
+```
+$ tree-sitter test
+```
+
+Many tests there were inherited from the grammar we forked. Many of those fail, and some shouldn't actually pass.
+
+In addition to a fixed test suite, we have `./tree-sitter-markdown-inline/scripts/shortcode_generator.py` to test the shortcode parsing subsystem specifically.
+It uses random testing to generate large numbers of shortcodes, calls `tree-sitter parse` on them, and checks if the output matches expectations.
+
+We use it to generate failing tests that are then fixed and added to the test suite (crates/tree-sitter-qmd/tree-sitter-markdown-inline/test/corpus/shortcodes.txt).
+At present time, we have generated over 10k random tests without failures.

crates/tree-sitter-qmd/notes.md renamed to docs/todo.md

@@ -1,6 +1,7 @@
-
 ## TODO
 
+- Allow attributes in editor marks (importantly for date/author)
+
 - attribute handling in ATX headers
 
 - equation handling
@@ -32,28 +33,3 @@
 
         - in addition, if we _do_ add support for in-block equation ids, we should consider that the output
           will not only need to exist for LaTeX, but will need to exist for html and typst as well.
-
-
-## notes
-
-- I really think Pandoc would benefit from the following rawblock extensions, and I 
-  wonder if we should try to adopt this into Quarto somehow:
-
-  - `{<format}` parses input from the rawblock as if it were Pandoc's `format`
-
-    - This would be, then, how code opts into Quarto's html-table-is-ast feature.
-
-  - 
-    ```
-    ::: {>format}
-    :::
-    ```
-
-    process this output as if it were markdown, but only produce output in `format`.
-
-    In Quarto, this is "just" syntax for
-
-    ```
-    ::: {.content-visible when-format="format"}
-    :::
-    ```

notes.md renamed to docs/wasm.md

@@ -1,19 +1,3 @@
-## Testing
-
-tree-sitter-markdown-inline has a tree-sitter test suite that can be run with
-
-```
-$ tree-sitter test
-```
-
-Many tests there were inherited from the grammar we forked. Many of those fail, and some shouldn't actually pass.
-
-In addition to a fixed test suite, we have `./tree-sitter-markdown-inline/scripts/shortcode_generator.py` to test the shortcode parsing subsystem specifically.
-It uses random testing to generate large numbers of shortcodes, calls `tree-sitter parse` on them, and checks if the output matches expectations.
-
-We use it to generate failing tests that are then fixed and added to the test suite (crates/tree-sitter-qmd/tree-sitter-markdown-inline/test/corpus/shortcodes.txt).
-At present time, we have generated over 10k random tests without failures.
-
 ## Wasm builds
 
 ```