Transforming Embedded HTML to the Target Format

[thekief]

I'm currently working on a project where sometimes the incoming data is a mixed bag of Markdown and some HTML sprinkled in-between. While this displays just fine when rendering the Markdown, the conversion strips various elements (</br>, <li>, etc.) that sometimes highly impact the layout of the output.

I looked into the discussions and found a few solutions to do the conversion of Markdown with HTML inside, but they were quite specialised. I was wondering, if there's a more generic approach.

Ideally this:

| Action                                                                                 | Erwartetes Result                                                 |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| Do something special                                                                   | Magic<br />Other things:<br /><ul><li>"just enjoy"</li></ul>      |
| Do something even more special<br />For that do<br /><ul><li>abc</li><li>def</li></ul> | Even more special magic.<br />Other things:<ul><li>asdf</li></ul> |

converts to

#figure(
  align(center)[#table(
      columns: (45.45%, 54.55%),
      align: (auto, auto),
      table.header([Action], [Result]),
      table.hline(),
      [Do something special Magic], [For that do \

        - abc
        - def
      ],
      [Do something even more special], [Even more special magic. \
        Other things:

        - asdf
      ],
    )],
  kind: table,
)

[rnwst]

You can see how your Markdown is parsed into pandoc's AST by outputting to the native format:

pandoc test.md -t native

You will notice that all your inline HTML is converted to RawInline HTML elements. When outputting to typst, these are not included because the output format is typst, not HTML. What you could do is convert to HTML first, and then from HTML to typst. That way the inline HTML elements are parsed into native AST elements rather than RawInlines during the second conversion:

pandoc test.md | pandoc -f html -t typst

This yields

#figure(
  align(center)[#table(
    columns: (56%, 43%),
    align: (auto,auto,),
    table.header([Action], [Erwartetes Result],),
    table.hline(),
    [Do something special], [Magic \
    Other things: \
    ],
    [Do something even more special \
    For that do \
    ], [Even more special magic. \
    Other things:],
  )]
  , kind: table
  )

I'm not familiar with typst, but I think this looks a little closer to your expected output.

[thekief]

Thank you for your input. One issue though, interpreting it as HTML removes the list items.

I looked a bit into the native output generated by pandoc -f gfm -tnative test.md and I was wondering this one thing:

pandoc already correctly recognises the HTML elements as such, and according to the gfm specification ¹ this Markdown flavour is supposed to support this mix of MD and HTML. Wouldn't it be useful to add a flag that tells it: "take HTML in Markdown and interpret it"? Wouldn't that fall into a similar category as native_{spans,divs} for HTML?

The bottom-right cell from the example above converts to, using pandoc -f gfm -tnative test.md:

[ Plain
    [ Str "Even"
    , Space
    , Str "more"
    , Space
    , Str "special"
    , Space
    , Str "magic."
    , RawInline (Format "html") "<br />"
    , Str "Other"
    , Space
    , Str "things:"
    , RawInline (Format "html") "<ul>"
    , RawInline (Format "html") "<li>"
    , Str "asdf"
    , RawInline (Format "html") "</li>"
    , RawInline (Format "html") "</ul>"
    ]
]

If a flag is not the way to go, would it possible to use a filter to do the conversion? I would need to fetch each cell, check the content for HTML inlines and do the conversion of the HTML only, if possible.

Footnotes

1. https://github.github.com/gfm/ (sadly there is nothing newer) ↩

[rnwst]

Unfortunately the list items cannot be represented in pandoc's AST, because they are inline elements and pandoc's BulletList type is a block-level element. If the surrounding elements were block-level too, this might work. So your proposed solution of parsing raw HTML in Markdown files as pandoc AST elements wouldn't work in this case.

Another thing I have tried is to convert the RawInlines to the typst format, via the following filter:

---@param rawInline RawInline
function RawInline(rawInline)
   local doc = pandoc.read(rawInline.text, 'html')
   logging.temp(rawInline.text)
   logging.temp(doc.blocks)
   local typst = pandoc.write(doc, 'typst')
   return pandoc.RawInline('typst', typst)
end

but that doesn't work either, because the individual RawInlines (e.g. <li>) cannot be converted to meaningful typst elements (the contents between the tags would be needed as well). You might be able to come up with a workable Lua filter though by experimenting a bit.

[thekief]

Thank you for the filter snippet, I think you brought me on the right track. I'll report back :D