Render markdown inside RTF or DOCX

[louisnichols]

Hi!

I have a tool (sort of a database) which is able to export content to RTF or DOCX.

Part of this content (but not all) is markdown.

Now, I can modify these exports to include separators for where markdown starts and ends. For example %%md ... %%.

Is there any way that I can use pandoc to parse such files and, where it meets text in between these separators, it renders just that part as markdown and saves it as a new file?

[rnwst]

That should be possible with a Lua filter, using the pandoc.read function to parse the Markdown elements within the document.
If you upload a simple example (RTF or DOCX), I may be able to have a play with this if I have some spare time.

[wlupton]

and/or add some debugging output to the filter. plug: https://github.com/pandoc-ext/logging.

[louisnichols]

Thanks. I'll try the logging, as the file is otherwise too big.

[louisnichols]

OK, so after a series of quite some weird realizations, I narrowed it down to this:

Two underlined words and a word in between {{...}} in the same block, with any combination of whitespace between them (I tried new lines with and without some additional spaces and it's the same). Actually, with any combination of characters between them, because additional text also results in the same thing.
If I delete any of these three words, the conversion works (sort of - details below). With all three, it doesn't.

And when I say conversion works, it's just that the script completes execution in a reasonable time. The outputs are strange.

creates

I am attaching the input file.
Debug_doc.docx

The command I use is pandoc Debug_doc.docx -L filter.lua -o DemoDoc.docx

The version I have:

pandoc -v 
pandoc 3.6.4
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: C:\Users\lpopescu\AppData\Roaming\pandoc
Copyright (C) 2006-2024 John MacFarlane. Web: https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

The filter in use is exactly the one suggested earlier. I'm including it below for reference

---Check whether the given block is a delimiter of the given type.
---@param block Block
---@param type ('opening' | 'closing')
---@return boolean
local function is_delimiter(block, type)
   if block.tag == 'Para' then
      local para = block
      ---@cast para Para
      if #para.content == 1 and para.content[1].tag == 'Str' then
         local str = para.content[1]
         ---@cast str Str
         if type == 'opening' and str.text == '§' then
            return true
         elseif type == 'closing' and str.text == '\\§' then
            return true
         end
      end
   end
   return false
end

---Blocks filter function.
---@param blocks (Blocks | Block[])
---@return Blocks
function Blocks(blocks)
   -- go back to front to avoid issues with changing indices
   for i = #blocks, 1, -1 do
      if is_delimiter(blocks[i], 'opening') then
         for j = i, #blocks do
            if is_delimiter(blocks[j], 'closing') then
               local md_blocks = pandoc.Blocks { table.unpack(blocks, i + 1, j - 1) }
               local inlines = pandoc.utils.blocks_to_inlines(md_blocks)
               local md = pandoc.write(pandoc.Pandoc(pandoc.Plain(inlines)), 'ansi', { columns = 99999 })
               local new_blocks = pandoc.read(md, 'markdown').blocks
               blocks = pandoc.Blocks { table.unpack(blocks, 1, i - 1) }
                  .. new_blocks
                  .. pandoc.Blocks { table.unpack(blocks, j + 1) }
               break
            end
         end
      end
   end
   return blocks
end

Any thoughts?

[rnwst]

Ok, I worked out what the problem is. The intermediate format should have been plain, not ansi, because ansi results in a bunch of ANSI escape codes when the text between the delimiters in your source document contains inline formatting such as bold text (such as in your example, where the dolor in braces is bold). Those ANSI escape codes is what you were seeing in the output ([22m[23m[...). To fix this, apply the following diff to the filter:

-               local md = pandoc.write(pandoc.Pandoc(pandoc.Plain(inlines)), 'ansi', { columns = 99999 })
+               local md = pandoc.write(pandoc.Pandoc(pandoc.Plain(inlines)), 'plain', { wrap_text = 'none' })

I have also updated the filter in the repo.

[louisnichols]

That works, thank you.

I had a bank holiday yesterday and only got to test this today. And it does the job.

However, I still have an issue with the reference doc.

I am trying to use the (newly) attached files with the command

pandoc Debug_doc.docx -L filter.lua --reference-doc=reference_doc.docx -o DemoDoc.docx

But I don't get the expected formatting for Tables. (I also don't get it for Headings, but that is a different topic which I'm still working on)

DemoDoc.docx
Debug_doc.docx
reference_doc.docx

Is this related to the fact that I use a filter?