Proposal: Add experimental_plugins (Syntax/AST/Render hooks) to markdown-rs

[jp-knj]

Motivation

As the number of plugins grows, the Rust↔JS round-trip cost starts to dominate and both throughput and stability become shaky. To reduce round-trips, we’d like to add a thin abstraction layer in markdown-rs so that popular transforms (slug, autolink, smartypants, GFM, etc.) can run natively in Rust while any remaining JS runs in a single batched pass.

Background context: see the Astro-side discussion about these tradeoffs in

• Adopt a Rust‑based Markdown/MDX Compiler as an Opt‑In withastro/roadmap#1195 (reply in thread))

Purpose

Under cfg(feature = "experimental_plugins"), introduce minimal traits and a light registry across three stages—Syntax, AST, and Render—so the “Host” (the runtime side) can manage registration order, priority, interop policy (GFM/MDX/Math/Frontmatter), and execution mode (Rust/WASM/JS-IPC). With the feature disabled, behavior is identical to today. We’ll start with AST hooks, then enable Syntax/Render as needed.

flowchart LR
  subgraph HOST["Host (runtime side)"]
    REG["PluginRegistry 
    (registration / order / priority)"]
    POL["Policy/Interop 
    (GFM / MDX / Math / Frontmatter)"]
    ADP["Adapters 
    (Rust / WASM / JS-IPC)"]
  end

  subgraph CORE["markdown-rs Core (minimal path)"]
    TK[Tokenizer / Parser]
    AST[MDAST]
    REN[Renderer]
  end

  IN[Markdown Input] --> TK --> AST --> REN --> OUT[HTML / HAST / SWC]
  REG --> POL --> ADP

  %% cross-boundary hooks
  ADP -. SyntaxPlugin (token edits) .-> TK
  ADP -. AstPlugin (tree transforms) .-> AST
  ADP -. RenderPlugin (node→output) .-> REN

  %% safe fallback on failure
  ADP -. runtime error .-> ERR{Plugin failure?}
  ERR -- yes --> FB[Fallback: literal text]
  FB --> OUT
  ERR -- no --> OUT

Loading

Summary

Core focuses on the minimal path Parser → MDAST → Renderer, and hooks connect across the Host/Core boundary as an optional layer. The Host supplies deterministic behavior via ordering and policy; on plugin failure we fall back to literal text so processing continues safely. This design minimizes round-trips, centralizes conflict resolution, and keeps production behavior predictable. We’ll demonstrate with small AST-first wins (e.g., heading ID, internal link normalization) before expanding to Syntax/Render.

Related: #52

[wooorm]

Hey!

For parser plugins, those are inline, here. That has to do with the language. Not that extendable compared to JS. And also bundle size often matters less.

For renderer, I think that’s just rehype/hast. No need to be here.

For the remaining one, ASTs. That’s here, indeed. But most AST stuff you mention is more about HTML (heading IDs).

What do you mean with link normalization?

[jp-knj]

Thanks a lot for the quick and clear reply. I really appreciate it.

I agree with your view: parser plugins stay inline in Rust, and renderer work lives in rehype/hast.

I’m only asking for two small hooks in the Markdown layer (not HTML):

1. Heading slugs (IDs) — let the host provide a slugger so each project can match its own rule (e.g. GitHub style).

2. Link normalization — by this I mean:

   • Reference labels: ignore case and collapse extra spaces so [Foo Bar] matches [foo bar]: ….
   • Link targets (before HTML): optionally add a base or locale, apply trailing-slash rules, change .md to .html, resolve ./ and ../, and optionally allow only safe schemes.

Which path fits this repo best?

A) Add tiny hooks in Core (the host provides the logic; defaults keep today’s behavior).
B) Keep Core as is; the host rewrites mdast→mdast outside Core.
C) Another approach you prefer (for example, keep it fully in rehype/hast).

If it helps, I can open a very small PR for A (just traits, defaults, and a couple of tests).

If it feels out of scope, please feel free to close PR or Discussion, and I’ll think about other approach.

Thanks again! Would you prefer A, B, or something else?