Further modularization?

[tarleb]

This question was triggered by three discussions that I had lately: in all three cases the subject were document conversions in the browser, and how to use pandoc components in a more modular fashion.

I'll use OJS as an example to illustrate the issue: most submissions happen in docx format, but having the document in HTML would be very convenient. Pandoc's docx converter is often "good enough" for this, so it would make sense to use that. However, shipping pandoc as part of OJS is not an option, as far as I've heard. It would be nice to have a small, specialized subset of pandoc, ideally compiled to wasm, so that it works in the browser.

One solution could be to pack selected readers and writers as separate packages, e.g., pandoc-docx-reader. The downside would be the increased maintenance burden that comes with more packages. We could write tools to make it easier, but still.

There might also be a need for more interfaces to make pandoc easier to use with other software, but I don't have a clear grasp yet of what would be needed. Issue #6611 might be an example, but there's probably more.

I hope to have some time to work on this over the course of next year.

Roping in some people who might have more to say: @cscheid @frederik @ronste

[jgm]

I'm all for making it possible to compile pandoc, or as much as possible of pandoc, to wasm.

Someone already had success doing this: have you studied this? https://github.com/haskell-wasm/pandoc-wasm
They report that the following changes were needed to allow compiling to wasm:

• Stripped out http client/server functionality
• Stripped out Lua support (Lua requires setjmp/longjmp which already work in wasi-libc to some extent, but that requires wasm exception handling feature which is not supported by wasmtime yet.)

Our current setup already separates all of the Lua functionality from the rest of pandoc, so that is covered. Stripping out HTTP support is more intrusive, since this is built into fetchItem in Text.Pandoc.Class, a central module.

[EDIT: forgot we'd already discussed this a bit in #10387 .]

[massifrg]

I'm all for making it possible to compile pandoc, or as much as possible of pandoc, to wasm.

I'm interested in calling pandoc wasm from the js/ts code of a web page.

My pundok-editor is electronjs-based, so it's a web app, where the server part is played by the "Main" process of electron.
The editor can only parse/write Pandoc's JSON format to/from Prosemirror objects, that are used to render the document.
When I need a conversion to/from another format, the web page code calls the "Main" process that spawns a pandoc process in the OS.
With a pandoc.wasm module, I could make the conversion in the browser, without spawning a process in the OS.

An example usage: in search and replace, you could specify the replacing text in markdown (or any other format) to get rich text features.
Suppose you want to search the regular expression \b(foo|bar|baz)\b, and replace it with _$1_ interpreted as markdown: the editor would do the replacement, so that foo becomes _foo_; then it would call pandoc.wasm to get the JSON version, before rendering it with Prosemirror.

The editor also lets you transform the current document applying a lua filter to it: that is done the same way, through a call to the "Main" process, that spawns a pandoc process running the filter.
A pandoc.wasm module would make it possible in the browser, though it could be limited to filters that don't require access to the file system.

[jgm]

As for splitting off pandoc-docx-reader, I shudder at the maintenance complexity of putting things into 50-100 different packages. Imagine making a small change to a central interface and needing to release all these things. But the above experience seems to show that we wouldn't need to do that to achieve wasm compilation.

[jgm]

On the other hand, a case could be made for creating a separate package for reading/writing docx, which defines docx-specific types and a parser and renderer for them. Pandoc could then use this package's API, but the package could also be used independently. This is the approach I've followed for some recent format additions: for djot and asciidoc, I created separate packages that each define an AST that is specific to the format; pandoc's readers just use these packages.

[tarleb]

I've started to do some work in that direction.