Add Markdoc/Jinja tag and HTML comment compatibility #15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jlevy
force-pushed
the
feature/markdoc-tag-compatibility
branch
from
19ea520 to
b4d2049
Compare
Improves Flowmark handling of template tags by: - Preserving newlines around Jinja/Markdoc tags - Preserving newlines around HTML comment tags - Keeping paired tags atomic during wrapping - Fixing adjacent tag spacing (no spurious spaces between paired tags) - Refactoring tag logic into separate modules This enables compatibility with Markdoc, Markform, WordPress Gutenberg, and other systems that use block-level template tags. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
jlevy
force-pushed
the
feature/markdoc-tag-compatibility
branch
from
b4d2049 to
64850f5
Compare
…tics - Add block_heuristics.py for table/list detection - Consolidate all tag-related logic in tag_handling.py: - Move normalize/denormalize adjacent tags functions - Move generate_coalescing_patterns and tag patterns - Add block content detection to preserve newlines in tags - Simplify text_wrapping.py by importing from tag_handling - Block heuristics only apply when tags are present to avoid changing normal markdown formatting behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Test that table rows and list items inside tags have their newlines preserved, and verify heuristics only apply when tags are present. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Marko-based parsing functions to block_heuristics.py for accurate
block content detection (lists, tables)
- Add blank line normalization between tags and block content to prevent
CommonMark lazy continuation issues
- Add post-processing to strip incorrect indentation from closing tags
that follow list items
- Add comprehensive unit tests for block content handling
- Update expected test files to reflect corrected output format
The key fix ensures closing tags like {% /field %} are not incorrectly
indented when they follow list items, which was caused by the Markdown
parser adding continuation indentation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Instead of just stripping indentation from closing tags, now also adds
a blank line before them when following block content. This is the
proper fix that makes the indentation naturally correct.
The key change is in `_fix_closing_tag_spacing()`:
- Detects closing tags ({% /, {# /, {{ /, <!-- /)
- Adds blank line before them if previous line is not empty or a tag
- Strips any incorrect indentation
This ensures output like:
```
{% field %}
- Option A
- Option B
{% /field %}
```
Instead of the previous incorrect output where the closing tag was
indented due to CommonMark lazy continuation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Added 4 new test functions covering: - Self-closing Jinja tags (break, include, set) - Self-closing HTML comment tags (note, TODO) - Jinja variable tags - Jinja comment tags These tests verify that standalone tags without closing counterparts are kept atomic and preserve newlines around them correctly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The bug was that line_wrap_by_sentence called wrap_paragraph_lines which does not call denormalize_adjacent_tags, causing spaces to be inserted between adjacent tags like --><!-- and not removed. Fix: Add denormalize_adjacent_tags call at the end of line_wrap_by_sentence. Added 5 comprehensive unit tests for adjacent tag handling: - test_adjacent_jinja_tags_no_space - test_adjacent_html_comment_tags_no_space - test_adjacent_jinja_variable_tags_no_space - test_adjacent_jinja_comment_tags_no_space - test_adjacent_tags_full_pipeline These tests cover normalize/denormalize directly, both line wrappers, and the full Markdown processing pipeline to catch future regressions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This fixes the bug where paragraph text between tags was incorrectly getting extra blank lines before closing tags. Now blank lines are only added when the previous content is block content (lists/tables). Changes: - Fix _fix_closing_tag_spacing() to check for block content, not all content - Add comprehensive unit tests for paragraph vs block content behavior - Add testdoc cases for tables with various tag types (Jinja, HTML, variables) - Add testdoc cases for paragraph-only content (should NOT get blank lines) - Add testdoc cases for self-closing tags with tables Test coverage added: - test_paragraph_text_no_extra_blank_lines() - test_list_content_gets_blank_lines() - test_table_content_gets_blank_lines() - test_mixed_content_blank_lines_correct() - test_closing_tag_spacing_function() - test_various_tag_types_with_tables() - test_paragraph_only_content_various_tags() Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…, add tests - Remove unused Marko-based functions (content_starts_with_block, content_is_block_element, normalize_block_content_in_tags) - Fix list item detection to require space/tab after marker per CommonMark spec - Previously "1.0 version" was incorrectly detected as a list item - Add CommonMark and GFM spec references in docstrings - Add inline unit tests for all heuristic functions - Update module docstring to explain why simple heuristics are appropriate Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update plan spec with final architecture (new modules: tag_handling.py, block_heuristics.py, protocols.py) - Mark all implementation phases as complete - Update "Known Limitations" to reflect auto-handling of block content spacing - Add Final Statistics section - Move completed spec from active/ to done/ Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds comprehensive support for Jinja/Markdoc tags (
{% %},{# #},{{ }}) and HTML comment tags (<!-- -->). This enables compatibility with Markdoc, Markform, WordPress Gutenberg, and similar templating systems.Key Features
Newline Preservation: Preserves newlines around tags - if a tag is on its own line in input, it stays on its own line in output
Paired Tags Stay Atomic: Tags like
{% field %}{% /field %}are kept together during wrapping, not broken across linesAdjacent Tags Stay Adjacent: No space inserted between
%}and{%or-->and<!--Block Content Handling: Blank lines are automatically added around tables and lists inside tags to prevent CommonMark lazy continuation issues
Paragraph Content Unchanged: Regular paragraph text between tags does NOT get extra blank lines - only block content (lists/tables) triggers blank line insertion
Architecture
tag_handling.py: Centralized tag detection, normalization/denormalization, coalescing patternsblock_heuristics.py: Detection of block content (tables, lists) for proper spacingprotocols.py: SharedLineWrapperprotocolTest Coverage
48 unit tests in
test_wrapping.pycovering:Comprehensive testdoc integration tests:
Test plan
make lintpasses with no warnings🤖 Generated with Claude Code