Docs for editorial marks (#62)

* document editorial marks, fix qmd roundtripping

* attempt to make claude remember red-light/green-light tdd

Author: cscheid

CLAUDE.md

@@ -2,11 +2,24 @@
 
 The main documentation for this repository is located at:
 [crates/quarto-markdown-pandoc/CLAUDE.md](crates/quarto-markdown-pandoc/CLAUDE.md)
+
+## **CRITICAL - TEST-DRIVEN DEVELOPMENT**
+
+When fixing ANY bug:
+1. **FIRST**: Write the test
+2. **SECOND**: Run the test and verify it fails as expected
+3. **THIRD**: Implement the fix
+4. **FOURTH**: Run the test and verify it passes
+
+**This is non-negotiable. Never implement a fix before verifying the test fails.**
+
+## General Instructions
+
 - in this repository, "qmd" means "quarto markdown", the dialect of markdown we are developing. Although we aim to be largely compatible with Pandoc, it is not necessarily the case that a discrepancy in the behavior is a bug.
 - the qmd format only supports the inline syntax for a link [link](./target.html), and not the reference-style syntax [link][1].
 - Always strive for test documents as small as possible. Prefer a large number of small test documents instead of small number of large documents.
 - When fixing bugs, always try to isolate and fix one bug at a time.
-- When fixing bugs using tests, run the failing test before attempting to fix issues. This helps ensuring that tests are exercising the failure as expected, and fixes actually fix the particular issue.
+- **CRITICAL - TEST FIRST**: When fixing bugs using tests, you MUST run the failing test BEFORE implementing any fix. This is non-negotiable. Verify the test fails in the expected way, then implement the fix, then verify the test passes.
 - If you need to fix parser bugs, you will find use in running the application with "-v", which will provide a large amount of information from the tree-sitter parsing process, including a print of the concrete syntax tree out to stderr.
 - use "cargo run --" instead of trying to find the binary location, which will often be outside of this crate.
 - when calling shell scripts, ALWAYS BE MINDFUL of the current directory you're operating in. use `pwd` as necessary to avoid confusing yourself over commands that use relative paths.

crates/quarto-markdown-pandoc/CLAUDE.md

@@ -25,13 +25,27 @@ Whenever you start working on a coding task, follow these steps:
 
 - Make a plan to yourself.
   - The plan should include adding appropriate tests to the test suite.
-  - Before implementing the feature, write the test that you think should fail, and ensure that the test fails the way you expect to!
+  - **FIRST**: Write the test that you think should fail
+  - **SECOND**: Run ONLY that test and verify it fails with the expected error
+  - **THIRD**: Implement the fix
+  - **FOURTH**: Run the test again and verify it now passes
 - Work on the plan item by item.
 - You are not done until the test you wrote passes.
 - You are not done until the test you wrote is integrated to our test suite.
 - If you run out of ideas and still can't make the test pass, do not erase the test. Report back to me and we will work on it together.
 - If in the process of writing tests you run into an unexpected parse error, store it in a separate file and report it to me. We're still improving the parser and it's possible that you will run into bugs.
 
+## BEFORE YOU IMPLEMENT ANY BUG FIX - MANDATORY CHECKLIST
+
+Before implementing ANY bug fix, you MUST complete this checklist in order:
+
+- [ ] Have you written the test?
+- [ ] Have you run the test and confirmed it fails?
+- [ ] Have you verified the failure is exactly what you expected?
+- [ ] Only after all three above are complete: proceed with implementation
+
+This is non-negotiable. Do not skip this process.
+
 # Error messages
 
 The error message infrastructure is based on Clinton Jeffery's TOPLAS 2003 paper "Generating Syntax Errors from Examples". You don't need to read the entire paper to understand what's happening. The abstract of the paper is:
@@ -65,12 +79,12 @@ The `quarto-markdown-pandoc` binary accepts the following options:
 - The qmd format only supports the inline syntax for a link [link](./target.html), and not the reference-style syntax [link][1].
 - Always strive for test documents as small as possible. Prefer a large number of small test documents instead of small number of large documents.
 - When fixing bugs, always try to isolate and fix one bug at a time.
-- When fixing bugs using tests, run the failing test before attempting to fix issues. This helps ensuring that tests are exercising the failure as expected, and fixes actually fix the particular issue.
+- **CRITICAL - TEST FIRST**: When fixing bugs using tests, you MUST run the failing test BEFORE implementing any fix. This is non-negotiable. Verify the test fails in the expected way, then implement the fix, then verify the test passes.
 - If you need to fix parser bugs, you will find use in running the application with "-v", which will provide a large amount of information from the tree-sitter parsing process, including a print of the concrete syntax tree out to stderr.
 - use "cargo run --" instead of trying to find the binary location, which will often be outside of this crate.
 - If you need to fix parser bugs, you will find use in running the application with "-v", which will provide a large amount of information from the tree-sitter parsing process, including a print of the concrete syntax tree out to stderr.
 - When fixing inconsistency bugs, use `pandoc -t json -i <input_file>` to get Pandoc's output, and `cargo run -- -t json -i <input_file>` to get our output.
-- When fixing roundtripping bugs, make sure to always add a roundtripping test to `tests/roundtrip_tests/qmd-json-qmd`.
+- **When fixing roundtripping bugs**: FIRST add the failing test to `tests/roundtrip_tests/qmd-json-qmd`, run it to verify it fails with the expected output, THEN implement the fix, THEN verify the test passes.
 - When I say "@doit", I mean "create a plan, and work on it item by item."
 - When you're done editing a Rust file, run `cargo fmt` on it.
 - If I ask you to write notes to yourself, do it in markdown and write the output in the `docs/for-claude` directory.

crates/quarto-markdown-pandoc/src/writers/qmd.rs

@@ -910,6 +910,30 @@ fn write_quoted(
     Ok(())
 }
 fn write_span(span: &crate::pandoc::Span, buf: &mut dyn std::io::Write) -> std::io::Result<()> {
+    let (id, classes, keyvals) = &span.attr;
+
+    // Check if this is an editorial mark span that should use decorated syntax
+    // These spans have exactly one class, no ID, and no key-value pairs
+    if id.is_empty() && classes.len() == 1 && keyvals.is_empty() {
+        let marker = match classes[0].as_str() {
+            "quarto-highlight" => Some("!! "),
+            "quarto-insert" => Some("++ "),
+            "quarto-delete" => Some("-- "),
+            "quarto-edit-comment" => Some(">> "),
+            _ => None,
+        };
+
+        if let Some(marker) = marker {
+            // Write using decorated syntax
+            write!(buf, "[{}", marker)?;
+            for inline in &span.content {
+                write_inline(inline, buf)?;
+            }
+            write!(buf, "]")?;
+            return Ok(());
+        }
+    }
+
     // Spans always use bracket syntax: [content]{#id .class key=value}
     // Even empty attributes should be written as [content]{} for proper roundtripping
     write!(buf, "[")?;

crates/quarto-markdown-pandoc/tests/roundtrip_tests/qmd-json-qmd/editorial_marks.qmd

@@ -0,0 +1 @@
+This paragraph has [!! a highlight], [++ an insertion], [-- a deletion], and [>> a comment] all together.

crates/quarto-markdown-pandoc/tests/roundtrip_tests/qmd-json-qmd/editorial_marks_individual.qmd

@@ -0,0 +1,7 @@
+Here is [!! a highlight].
+
+Here is [++ an insertion].
+
+Here is [-- a deletion].
+
+Here is [>> a comment].

docs/syntax/editorial-marks.qmd

@@ -0,0 +1,142 @@
+---
+title: "Editorial Marks"
+---
+
+## Overview
+
+Editorial marks allow you to annotate your text with suggestions, highlights, and comments during the editing and review process. In `quarto-markdown`, these marks use a special bracket syntax and are converted to Pandoc AST spans with specific CSS classes that can be styled or processed by filters.
+
+## Syntax
+
+Editorial marks use a square bracket syntax with special marker characters:
+
+- `[!! text]` - Highlight
+- `[++ text]` - Insertion suggestion
+- `[-- text]` - Deletion suggestion
+- `[>> text]` - Comment
+
+The opening bracket is followed by two marker characters, then a space, and then the content to be marked. The mark is closed with a closing bracket.
+
+## Conversion to Pandoc AST
+
+When converted to Pandoc AST, editorial marks become `Span` elements with specific CSS classes:
+
+- `[!! ...]` → `Span ("", ["quarto-highlight"], []) [...]`
+- `[++ ...]` → `Span ("", ["quarto-insert"], []) [...]`
+- `[-- ...]` → `Span ("", ["quarto-delete"], []) [...]`
+- `[>> ...]` → `Span ("", ["quarto-edit-comment"], []) [...]`
+
+These classes can be used to apply styling in HTML output or to process the marks with Pandoc filters.
+
+## Examples
+
+### Highlight
+
+Use `[!! text]` to highlight important sections that need attention:
+
+This paragraph contains [!! an important section] that should be reviewed.
+
+```markdown
+This paragraph contains [!! an important section] that should be reviewed.
+```
+
+### Insertion Suggestion
+
+Use `[++ text]` to suggest text that should be added:
+
+The document needs [++ additional context here] to be complete.
+
+```markdown
+The document needs [++ additional context here] to be complete.
+```
+
+### Deletion Suggestion
+
+Use `[-- text]` to suggest text that should be removed:
+
+This sentence has [-- unnecessary words] that can be deleted.
+
+```markdown
+This sentence has [-- unnecessary words] that can be deleted.
+```
+
+### Comment
+
+Use `[>> text]` to add editorial comments or notes:
+
+The conclusion [>> needs more supporting evidence] requires revision.
+
+```markdown
+The conclusion [>> needs more supporting evidence] requires revision.
+```
+
+### Multiple Marks in One Paragraph
+
+You can use multiple editorial marks in a single paragraph:
+
+This paragraph has [!! a highlight], [++ an insertion], [-- a deletion], and [>> a comment] all together.
+
+```markdown
+This paragraph has [!! a highlight], [++ an insertion], [-- a deletion], and [>> a comment] all together.
+```
+
+### Nested Formatting
+
+Editorial marks can contain other inline formatting:
+
+The section needs [++ **bold text** and *italic text*] to emphasize the point.
+
+```markdown
+The section needs [++ **bold text** and *italic text*] to emphasize the point.
+```
+
+## Use Cases
+
+Editorial marks are particularly useful for:
+
+- **Collaborative editing**: Team members can suggest changes without directly modifying the text
+- **Review workflows**: Reviewers can highlight issues and suggest improvements
+- **Track changes**: Similar to word processor track changes, but in plain text
+- **Editorial comments**: Editors can leave notes without disrupting the flow of the document
+
+## Processing Editorial Marks
+
+Since editorial marks are converted to spans with CSS classes, you can:
+
+1. **Style them in HTML output**: Use CSS to visually distinguish different mark types
+2. **Process with Lua filters**: Write Pandoc Lua filters to convert marks to other formats
+3. **Accept/reject suggestions**: Build tools to process insertion and deletion suggestions
+4. **Generate reports**: Extract all comments and highlights for review
+
+### Example CSS Styling
+
+```css
+.quarto-highlight {
+  background-color: yellow;
+}
+
+.quarto-insert {
+  color: green;
+  text-decoration: underline;
+}
+
+.quarto-delete {
+  color: red;
+  text-decoration: line-through;
+}
+
+.quarto-edit-comment {
+  color: blue;
+  font-style: italic;
+}
+```
+
+## Availability
+
+Editorial marks are currently only available when using `quarto-markdown-pandoc`. This syntax will be integrated into Quarto's markdown processing in future releases.
+
+## Limitations
+
+- Editorial marks only support inline content, not block-level elements
+- The marker characters (`!!`, `++`, `--`, `>>`) must be followed by a space
+- Editorial marks cannot be nested within each other

docs/syntax/index.qmd

@@ -11,4 +11,5 @@ The features documented here are currently under development. The syntax and beh
 ## Available Features
 
 - [Definition Lists](definition-lists.qmd) - Create definition lists using an embedded markdown DSL
+- [Editorial Marks](editorial-marks.qmd) - Annotate text with highlights, insertions, deletions, and comments
 - [Footnotes](footnotes.qmd) - Add footnotes with inline or fenced block syntax