Typesetting Pandoc JSON files directly with ConTeXt; an XML equivalent of JSON and native

[massifrg]

Hello,
I wrote a module for ConTeXt that lets you typeset Pandoc JSON files directly with ConTeXt.

The goal is to have all the information of a JSON file available in ConTeXt.

In particular, the module converts Pandoc JSON into an equivalent XML, something like this:

<Pandoc api-version="1,23,1">
<meta>
...
</meta>
<blocks>
<Para><Quoted quote-type="DoubleQuote">Hello, <Emph>world</Emph>!</Quoted></Para>
</blocks>
</Pandoc>

The conversion happens in memory, and it is pretty straightforward, because the tags have the same names of Pandoc items (see the project README).

The project has also a script to convert JSON files to XML files.

ConTeXt has great tools to typeset XML, that's why I chose it, instead of directly parsing JSON or markdown with Lua+LPEGs.

I also started writing a Haskell Writer for Pandoc for that XML format, but I'm a newbie in Haskell, so don't expect anything soon.

Also, before delving into that, I'd like to see if there's any interest in such a format, that would be a third form of Pandoc AST, beside native and JSON.

[jgm]

Well, it's much more readable than pandoc's JSON serialization, and unlike the native serialization, it's readable by standard tools. So I like the idea, personally. Drawback: adds complexity, possibly more things to change when the AST changes. We would need both a reader and a writer. If they could be implemented in a way that just piggybacked off the FromJSON / ToJSON instances for the types, then that would be ideal, because changing these instances would automatically affect both the JSON and XML serializations, and presumably the code could be quite short.

[massifrg]

FromJSON/ToJSON use an intermediate Value structure. https://hackage.haskell.org/package/aeson-2.2.3.0/docs/Data-Aeson.html#t:Value So code that converts a Value to XML would suffice for the writer, and code that converts XML to a Value would suffice for the reader.

I understand your point, but after trying to write down some haskell code to make a "rote" conversion from JSON to XML, I'm finding it very difficult.
Most of it is due to my very limited knowledge of Haskell, but perhaps there's still a problem that is independent from that: data structures that have no explicit name, like list items, DefinitionList terms and defs, or colspecs in tables.

I'm afraid the resulting XML would be less meaningful; unnamed values would likely result in XML elements with generic names like "array", "list", "object", etc.
Unless you discriminate on the "t" field of the parent node, making it less automatic.
Discriminating on the "t" field is more like a traditional fromXML/toXML approach, and this would pull back the complexity you wanted to keep out.

[jgm]

Agreed, you'd need to look at the t field and come up with meaningful names. Going via Aeson Value might still reduce some complexity, though.

[massifrg]

Just an update after reading @andrewufrank's reply on issue #10651.

I looked into aeson and now I understand better why you'd go for a XML -> JSON -> AST -> JSON -> XML workflow.

The reader side (XML -> JSON) looks easier, since it's parsing XML with a standard library and creating a JSON.

On the writer side (JSON -> XML), the main problem is how to decide what goes into XML attributes and what goes instead into the children elements of an element.
Perhaps it's possible to derive a generic rule, in a way similar to how aeson maps obiects to JSON structures pretty automatically.

I'd say simple, unformatted strings and numbers should go into XML attributes, e.g.:

• the level of a Header
• the Attr data
• MathType, QuoteType, raw's format, ordered lists attributes
  More complex data structures should be expressed as children elements.

Another problem is attributes' names. In Pandoc JSON there's no name, e.g., for the first element of a Header tuple, you just know that it's a number, but you can't figure out that it's called "level" automatically.

[massifrg]

We use xml-conduit for XML parsing (and it also supports rendering). https://hackage.haskell.org/package/xml-conduit-1.9.1.4/docs/Text-XML.html So two way conversion between an xml-conduit Node and an aeson Value would probably be sufficient for this.

I started working on the XML writer, following your advice.

Pandoc uses Text.XML.Light, not Text.XML, am I right?
Anyway, I'm starting with Text.XML.Light.

BTW, I found @tarleb 's "short guide to pandoc's sources" in the doc directory of the sources, that helped me.
I found it releaving that he says in the guide that pandoc's sources were his first exposure to Haskell, because I'm in a similar situation now.
@jgm: what's the best way to ask for help about working on pandoc's haskell sources?

[jgm]

Pandoc source code includes a special version of xml-light (in the xml-light tree) that replaces the old xml-light parser with xml-conduit's much better parser, but then changes everything to xml-light types so that the legacy code could stay the same. So, basically we're using xml-conduit, and for anything new it may make sense to use that directly rather than via our special xml-light.

For an XML writer, though, you may not need either (those are for parsing XML). What we use for XML writers is Text.Pandoc.XML, which contains functions for rendering XML.

But I guess if you're defining a two-way conversion between Aeson and (say) xml-conduit's types, then it would make sense to use xml-conduit's renderer rather than Text.Pandoc.XML. This is probably also faster.