Skip to content

fix(parser): preserve Markdown code fence boundaries - #356

Open
er-s-an wants to merge 1 commit into
HKUDS:mainfrom
er-s-an:codex/preserve-markdown-code-fences
Open

fix(parser): preserve Markdown code fence boundaries#356
er-s-an wants to merge 1 commit into
HKUDS:mainfrom
er-s-an:codex/preserve-markdown-code-fences

Conversation

@er-s-an

@er-s-an er-s-an commented Sep 7, 2026

Copy link
Copy Markdown

Description

Keep Markdown code examples literal until a matching closing fence is reached.

After #352, the direct Markdown parser records every opening fence as only three characters. A triple-backtick line inside a four-backtick code block therefore ends the block early. A fence-like line with trailing text (for example, three backticks followed by python) also incorrectly closes it.

As a result, headings and local image references inside the example become document heading/image blocks, while the actual closing fence can hide real headings/images that follow it. This breaks the existing promise that fenced content is not interpreted; it is not a request for broader Markdown support.

Changes Made

  • Retain the opening fence's full length.
  • Close only on the same marker, with at least that length and no non-whitespace suffix, following the CommonMark closing-fence rules.
  • Add 10 parameterized cases covering shorter fences, trailing info text, mixed marker types, longer valid closers with whitespace, and an unterminated block. The file-based regression also checks that a real image reference after the valid closer still becomes an image block.

Validation

  • Before the production change: 6 new regression cases failed; 17 cases passed in tests/test_plain_text_direct.py.
  • After the change: python -m pytest tests/ -q406 passed on Python 3.12.13.
  • The full suite also passed with directory-based coverage (--cov=raganything --cov-branch).
  • Ruff 0.6.4 check (--ignore=E402) and format check passed for both changed files; git diff --check passed.

Related Issues and Limitations

Follow-up to the fenced-content behavior introduced in #352. I did not find an existing issue or PR for this edge case.

This is an offline parser regression, not a live OCR/VLM or ingestion-performance test. The image fixture only needs to exist because this parsing path resolves references without decoding images. No new dependency, public API, or other Markdown syntax handling is introduced. Python 3.10/3.11 are left to CI. An initial coverage invocation targeting the dotted module name triggered import-time errors; the directory-targeted invocation above completed successfully.

Prepared with AI assistance; the changed paths and local test results were checked before submission.

Signed-off-by: er-s-an <3137612974@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant