Improve `pandoc.scaffolding` section in 'Pandoc Lua Filters' docs (RE: Custom Writers)

[0xdevalias]

In the spirit of:

• https://pandoc.org/CONTRIBUTING.html

  • Write or improve documentation. If you ran into a problem which took more time to figure out than expected, please consider to save other users from the same experience. People writing the documentation tend to lack an outside view, so please help provide one. Good documentation is both difficult and extremely important.

Describe your proposed improvement and the problem it solves.

In reading through the Lua Filters documentation recently, I found this section on pandoc.scaffolding for Writers:

• https://pandoc.org/lua-filters.html#module-pandoc.scaffolding

  • Scaffolding for custom writers.

But the Fields section is empty (unsure if that is how it is meant to be, but perhaps adding some text to say that explicitly if so to remove ambiguity):

• https://pandoc.org/lua-filters.html#pandoc.scaffolding-fields

The Writer section is also fairly light on details:

• https://pandoc.org/lua-filters.html#pandoc.scaffolding.Writer

  • An object to be used as a Writer function; the construct handles most of the boilerplate, expecting only render functions for all AST elements (table)

In doing some googling, I also came across these pages:

• https://pandoc.org/custom-writers.html
  • https://pandoc.org/custom-writers.html#example-modified-markdown-writer
  • https://pandoc.org/custom-writers.html#reducing-boilerplate-with-pandoc.scaffolding.writer
• https://pandoc.org/chunkedhtml-demo/3.3-general-writer-options.html
• https://pandoc.org/chunkedhtml-demo/15-custom-readers-and-writers.html
  • https://github.com/jgm/djot/blob/2336a695176d9ef15661faa5a425d9a372187db9/djot-writer.lua

It would be good to fill out this section of the docs more, even if that is just adding some cross-links to the existing pages on Custom Writers/etc.

Describe alternatives you've considered.

I'm aware of this issue/PR, but the nuance is different:

• Restructure Lua and JSON filter docs #8750
  • Simplify and reorganize Lua filter introduction #9106

This docs issue is also unrelated, except for improving the docs UX for new users in general:

• Improve cross-linking of types in 'Pandoc Lua Filters' docs #10529

[0xdevalias]

The following is a bit of a mind dump of my process/thoughts/random ideas of what would make this easier as a new user wanting to make a custom writer:

Have been trying to play with pandoc.scaffolding.Writer a little more this afternoon, and it seems like we basically just need to implement handlers for each of the node types; but we have the flexibility to override functions if when needed:

• https://pandoc.org/custom-writers.html#reducing-boilerplate-with-pandoc.scaffolding.writer

  • The pandoc.scaffolding.Writer structure is a custom writer scaffold that serves to avoid common boilerplate code when defining a custom writer. The object can be used as a function and allows to skip details like metadata and template handling, requiring only the render functions for each AST element type.

    The value of pandoc.scaffolding.Writer is a function that should usually be assigned to the global Writer

  • All predefined functions can be overwritten when needed.

I may have missed it when skimming around, but I have been struggling to figure out how to see what the default functions this creates actually are:

• I tried print(Writer), but it just gives me table: 0x600000323f00
• I tried looking in the code, but that wasn't very clear to me
  • https://github.com/jgm/pandoc/blob/56625255b36f1d675f42e5785ff5ffbff9cb7418/pandoc-lua-engine/src/Text/Pandoc/Lua/Module/Scaffolding.hs

I think it would be helpful to expand the documentation to:

• explain more about what functions are generated/handled by this helper method
• perhaps also to give an example of how to inspect those ourselves (in case the documentation gets out of sync with the implementation at some point)
• an example of how to override one of the default functions

The next thing I noticed was that if we just create a writer with Writer = pandoc.scaffolding.Writer and try to run it, we will immediately get errors about not implemented handlers for the various AST nodes. While this makes sense, as it is a 'custom writer' that we haven't customised; it might be nice if there was an option to allow it to delegate to a specified writer implementation for the nodes by default, and then we could just override the ones we want to change. For example, I would like a custom writer that is mostly gfm, but that I tweak for Emph / BulletList / etc (eg. #10527, #10479, etc).

But since that's not currently a feature (to my knowledge), I next attempted to find a list of AST nodes that I needed to implement, and from a quick attempt to skim around the lua docs, I couldn't find a neat canonical list (#3262) I'm sure I could iterate through all the headings to build one, but really I wanted to be able to copy/paste a paragraph or so and reformat it into lua functions; or even better would be if there was a default set of functions I could copy from the lua filter docs/etc that would cover all of the boilerplate of setting up a 'delegating custom writer'; or perhaps this could also be 'baked in' to pandoc by adding a canonical example 'delegating custom writer' to the user data directory, that we could then access via something like pandoc --print-default-data-file custom/delegating-custom-writer-example.lua; or perhaps some combination of these methods.

Luckily I remember that I was reading through some of the Haskell source for AST nodes the other day, and copied a list of them into my gist, so I could use that as a starting reference; but I don't remember where I actually found that originally. #3288 seems to allude to it, and links to this code that seems to define relevant things, though its not immediately clear to me if thats everything I would need to implement or not (and as a new user, I wouldn't want to have to dig through the source to find these sorts of answers):

https://github.com/jgm/pandoc-types/blob/dc56b9a9678843649a6b1b50d255cc689fba4412/src/Text/Pandoc/Definition.hs#L315-L336

https://github.com/jgm/pandoc-types/blob/dc56b9a9678843649a6b1b50d255cc689fba4412/src%2FText%2FPandoc%2FDefinition.hs#L269-L303

I was originally looking at the old djot implementation for inspiration, and noticed this pattern that seems to allow running something for unimplemented entries; I wonder if we could do similar here to hack some kind of 'delegating custom writer':

• https://github.com/jgm/djot/blob/2336a695176d9ef15661faa5a425d9a372187db9/djot-writer.lua#L95-L107

Blocks = {}
Blocks.mt = {}
Blocks.mt.__index = function(tbl,key)
  return function() io.stderr:write("Unimplemented " .. key .. "\n") end
end
setmetatable(Blocks, Blocks.mt)

Inlines = {}
Inlines.mt = {}
Inlines.mt.__index = function(tbl,key)
  return function() io.stderr:write("Unimplemented " .. key .. "\n") end
end
setmetatable(Inlines, Inlines.mt)

• https://www.lua.org/pil/13.html

  • 13 – Metatables and Metamethods

  • Metatables allow us to change the behavior of a table.

  • We can use setmetatable to set or change the metatable of any table

  • It seems there is also a getmetatable
• https://www.lua.org/pil/13.3.html

  • 13.3 – Library-Defined Metamethods

  • It is a common practice for some libraries to define their own fields in metatables.

  • The tostring function provides a typical example. As we saw earlier, tostring represents tables in a rather simple format:

    print({})      --> table: 0x8062ac0
    

    (Note that print always calls tostring to format its output.) However, when formatting an object, tostring first checks whether the object has a metatable with a __tostring field. If this is the case, tostring calls the corresponding value (which must be a function) to do its job, passing the object as an argument. Whatever this metamethod returns is the result of tostring.

  • The setmetatable/getmetatable functions use a metafield also, in this case to protect metatables. Suppose you want to protect your sets, so that users can neither see nor change their metatables. If you set a __metatable field in the metatable, getmetatable will return the value of this field, whereas setmetatable will raise an error

• https://www.lua.org/pil/13.4.html

  • Table-Access Metamethods

  • But Lua also offers a way to change the behavior of tables for two normal situations, the query and modification of absent fields in a table.

  • https://www.lua.org/pil/13.4.1.html

    • 3.4.1 – The __index Metamethod

    • when we access an absent field in a table, the result is nil. This is true, but it is not the whole truth. Actually, such access triggers the interpreter to look for an __index metamethod: If there is no such method, as usually happens, then the access results in nil; otherwise, the metamethod will provide the result.

  • https://www.lua.org/pil/13.4.2.html

    • 13.4.2 – The __newindex Metamethod

    • The __newindex metamethod does for table updates what __index does for table accesses. When you assign a value to an absent index in a table, the interpreter looks for a __newindex metamethod: If there is one, the interpreter calls it instead of making the assignment.

    • The combined use of __index and __newindex metamethods allows several powerful constructs in Lua, from read-only tables to tables with default values to inheritance for object-oriented programming.

  • https://www.lua.org/pil/13.4.4.html

    • 13.4.4 – Tracking Table Accesses

    • Both __index and __newindex are relevant only when the index does not exist in the table. The only way to catch all accesses to a table is to keep it empty. So, if we want to monitor all accesses to a table, we should create a proxy for the real table. This proxy is an empty table, with proper __index and __newindex metamethods, which track all accesses and redirect them to the original table.

From reading deeper about mettables and metmethods like __index, it sounds like that would probably be a good way to implement a 'delegating custom writer'; and the 'tracking table access' might give us a way to see everything being called (as a way of seeing what functions exist maybe?)

I thought I tried this before.. but apparently not, as it seems to work somewhat:

Writer = pandoc.scaffolding.Writer

print("Writer:")
for n in pairs(Writer) do
  elem = Writer[n]

  print(n, type(n), type(elem))

  if type(elem) == "table" then
    print("--Foo--")
    for m in pairs(elem) do
      print("  ", m, type(m))
    end
    print("--Bar--")
  end
end

Outputting:

Writer:
Inlines	string	function
Blocks	string	function
Pandoc	string	function
Inline	string	table
--Foo--
--Bar--
Block	string	table
--Foo--
--Bar--

I can also see that directly calling something like Writer.Inline.Plain() will trigger the function not existing, which I assume means the 'handle default' metatable functionality is implemented on Writer.Inline and Writer.Block; which we could override:

Writer = pandoc.scaffolding.Writer

Writer.Inline.Plain()

Error running Lua:
No render function for Block value 'Plain';
define a function `Writer.Block.Plain` that returns a string or Doc.
stack traceback:
	...lias/.local/share/pandoc/custom/gfm_devalias_writer4.lua:27: in main chunk

Next I would just need to figure out if/how I can get access to the existing writer's handlers from lua.. or if not, a method for doing similar (eg. calling write within each handler)

I think that's the end of my mind dump thoughts for now.

[jgm]

I couldn't find a neat canonical list

We should have that. We used to have a sample old-style custom writer in the source tree, but it seems to have been deleted in 79d6b45 . This contained all of the things you'd need to implement.

The manual now points you to djot-writer.lua as a canonical example.

it might be nice if there was an option to allow it to delegate to a specified writer implementation for the nodes by default, and then we could just override the ones we want to change.

I've talked to @tarleb about this idea. We agree it would be great to have this, but it's not possible without some architectural changes.

[0xdevalias]

The manual now points you to djot-writer.lua as a canonical example.

I have been looking at that, and it's been helpful, but a few points:

• It's no longer part of the repo:

  • This file was deleted (on Jun 5, 2023)

  • 
  • https://github.com/jgm/djot/commits/main/djot-writer.lua
• The last version there is built in seemingly the old style, and not using pandoc.scaffolding.Writer, so while it's a good general reference, it's more verbose than it needs to be with the modern APIs.

---

We agree it would be great to have this, but it's not possible without some architectural changes.

nods makes sense, and I figured that was likely the case. Just wanted to make sure to raise it all the same.

[0xdevalias]

It's no longer part of the repo

Actually, looking at the commit history, seems there is a jgm/djot.lua repository as well:

• https://github.com/jgm/djot/commits/main/djot-writer.lua
  • jgm/djot@ebb4b71

    • Move all the Lua code to new djot.lua repository.

    • Create djot-writer.lua djot#233
      • Create djot-writer.lua djot#233 (comment)

        • Oh, I missed that this was djot and not djot.lua. Well, I don't think this really belongs here.
          We could think about replacing djot-writer.lua in the jgm/djot.lua repository with this one (which just adds a few things).
          And then changing the link from the pandoc documentation to point to that repo...

      • Create djot-writer.lua djot#233 (comment)

        • I for once think this is not a good idea: iiuc, djot is a moving target; maybe renaming it to oldjot-writer.lua and placing somewhere in either tree would be enough; but pointing to the right repo in manual.txt would be certainly welcome.

      • Create djot-writer.lua djot#233 (comment)

        • I meant to say that merging this lua filter with the main djot-writer.lua in /djot.lua seemed to me like a bad idea, for I fear it may break something in the /djot.lua program, and it may continuously fall way behind the upgrades on /jdot.js, which you say is the main development branch.

          So I thought renaming this filter (to oldjot-writer.lua for instance) and installing it in any other project (what I meant by "tree", as if each project/branch was a directory tree containing many files), such as /djot would avoid the versioning conflict. In any event, /pandoc/manual.txt needs updating.

      • Create djot-writer.lua djot#233 (comment)

        • In fact, I believe that the best place for [ol|sample]djot-writer.lua would be pandoc/data, (i.e. jgm/pandoc/tree/main/data), along with creole.lua, the sample custom reader.

• jgm/djot.lua:
  • https://github.com/jgm/djot.lua/blob/main/djot.lua

    • Parse and render djot light markup format. See https://djot.net.

  • https://github.com/jgm/djot.lua/blob/main/djot-reader.lua
    • Custom Reader
  • https://github.com/jgm/djot.lua/blob/main/djot-writer.lua
    • Custom Writer

Looking at that timeline and versions of the code:

• The code was moved to jgm/djot.lua in jgm/djot@ebb4b71.
• The last update to djot-writer.lua before that was in jgm/djot@239969f
  • https://github.com/jgm/djot/blob/239969fd84b6406b1f29e3514186667a5ac6c1cc/djot-writer.lua
• Create djot-writer.lua djot#233 was merged in jgm/djot@2336a69, and then reverted in jgm/djot@f22711a
  • https://github.com/jgm/djot/blob/2336a695176d9ef15661faa5a425d9a372187db9/djot-writer.lua

Comparing those 2 versions shows the diff is quite minimal:

• jgm/djot@239969fd84b6406b1f29e3514186667a5ac6c1cc...2336a695176d9ef15661faa5a425d9a372187db9#diff-5db5942876

--- -- Last version before moved to jgm/djot.lua: http…
+++ -- Version from https://github.com/jgm/djot/pull/2…
@@ -1,5 +1,9 @@
--- Last version before moved to jgm/djot.lua: https://github.com/jgm/djot/blob/239969fd84b6406b1f29e3514186667a5ac6c1cc/djot-writer.lua
+-- Version from https://github.com/jgm/djot/pull/233 merged in https://github.com/jgm/djot/blob/2336a695176d9ef15661faa5a425d9a372187db9/djot-writer.lua
 -- custom writer for pandoc
+-- example of upgrading of old style writer (pandoc <3.0)
+-- into new style (pandoc >= 3.0)
+-- see end of file for modified Writer function incorporating code snippet
+-- @@jarnosz at github 2023-06-01
 
 local unpack = unpack or table.unpack
 local format = string.format
@@ -439,6 +443,13 @@
 end
 
 function Writer (doc, opts)
+-- begin patch
+-- function Writer (doc, opts)
+  PANDOC_DOCUMENT = doc
+  PANDOC_WRITER_OPTIONS = opts
+  loadfile(PANDOC_SCRIPT_FILE)()
+-- return pandoc.write_classic(doc, opts) 
+-- end patch
   local d = blocks(doc.blocks, blankline)
   local notes = {}
   for i=1,#footnotes do

And diffing the last non-reverted version of djot-writer.lua in jgm/djot before transitioning with the latest version in jgm/djot.lua shows also minimal recent changes:

--- -- Last version before moved to jgm/djot.lua: http…
+++ -- Latest version on jgm/djot.lua (from 2023-11-03…
@@ -1,4 +1,4 @@
--- Last version before moved to jgm/djot.lua: https://github.com/jgm/djot/blob/239969fd84b6406b1f29e3514186667a5ac6c1cc/djot-writer.lua
+-- Latest version on jgm/djot.lua (from 2023-11-03) https://github.com/jgm/djot.lua/blob/7d1fbbb347c0f35b92590ed799501e8ba7115156/djot-writer.lua
 -- custom writer for pandoc
 
 local unpack = unpack or table.unpack
@@ -439,12 +439,18 @@
 end
 
 function Writer (doc, opts)
+  PANDOC_WRITER_OPTIONS = opts
   local d = blocks(doc.blocks, blankline)
   local notes = {}
   for i=1,#footnotes do
     local note = hang(blocks(footnotes[i], blankline), 4, concat{format("[^%d]:",i),space})
     table.insert(notes, note)
   end
-  return layout.render(concat{d, blankline, concat(notes, blankline)}, opts.columns)
+  local formatted = concat{d, blankline, concat(notes, blankline)}
+  if PANDOC_WRITER_OPTIONS.wrap_text == "wrap-none" then
+    return layout.render(formatted)
+  else
+    return layout.render(formatted, opts.columns)
+  end
 end

Since there are minimal differences across all 3 versions, I will just look at/comment on the latest version in jgm/djot.lua as a source of example for custom writers:

• It still seems to be largely written using the old style custom writer, and seems to sort of half implement the changes for 3.x (namely, setting PANDOC_WRITER_OPTIONS, but doesn't use pandoc.write_classic/etc):
  • https://pandoc.org/custom-writers.html#changes-in-pandoc-3.0
  • It seems there was some effort to change bits of this ages back, but seemingly not everything:
    • writer: Use options passed to the Writer function djot#56 (jgm/djot@94d1a20)
  • And then this was added in jgm/djot.lua@c87a4b1

    • As specified in the Pandoc 3.0 release PANDOC_WRITER_OPTIONS is not longer defined by pandoc. This fixes the writer.

• It's not specifically using pandoc.scaffolding.Writer, but from my understanding of how it works, I think it's implemented in basically the same way, just explicitly within the custom writers lua code itself:
  • eg. This Blocks / Inlines + setmetatable implementation appears to be equivalent to pandoc.scaffolding.Writer's implementation of Writer.Block and Writer.Inline (note the non-plural in the newer name change)

    • The render functions for Block and Inline values can then be added to Writer.Block and Writer.Inline, respectively. (Ref)

  • This inlines / blocks implementation appears to be equivalent to pandoc.scaffolding.Writer's implementation of Writer.Blocks and Writer.Inlines

    • the functions Writer.Blocks and Writer.Inlines can be used to render lists of elements, and Writer.Pandoc renders the document’s blocks. (Ref)

  • The Writer function is roughly equivalent to pandoc.scaffolding.Writer's main generated Writer function (aka: the result from calling it: Writer = pandoc.scaffolding.Writer)
  • etc

Based on this discovery, a few things that would probably be worth doing:

• Update the doc/lua-filters.md docs
  • Potentially could update the Global variables section that mentions PANDOC_WRITER_OPTIONS + the 'deprecated in 3.0' notes to link through to the 'Changes in pandoc 3.0' section of doc/custom-writers.md
    • I'm not sure if this also changed in 3.0, but given it's mentioned in the docs on how to convert a classic script, it may also be worth adding similar notes to the PANDOC_SCRIPT_FILE global variable; and maybe adding PANDOC_DOCUMENT with a similar deprecated note (for anyone looking at the docs for 2.x versions still)

    • PANDOC_SCRIPT_FILE : The name used to involve the filter. This value can be used to find files relative to the script file. This variable is also set in custom writers.

  • Maybe update any mention of 'custom writers' to link to doc/custom-writers.md
  • Maybe update write_classic's docs to link through to doc/custom-writers.md
    • And similar for write
    • And read should probably link through to docs/custom-readers.md
  • Update 'Module pandoc.scaffolding section with more complete details/information, including links to doc/custom-writers.md, and maybe also to docs/customizing-pandoc.md
  • Potentially add a new brief/reference section that gives a very quick high level summary of custom readers/writers/etc, and then links to the more detailed sections of the rest of the docs.
• Update the doc/custom-writers.md docs
  • Do something to include/reference the details made in the manual:
    • 79d6b45
    • Also maybe expand on the patterns used in this script, and how they align to pandoc.scaffolding.Writer, and maybe even an example of how to upgrade those patterns/etc
  • Update the opening paragraph to link to the classic custom writers section:

    • Pandoc also supports "classic" custom writers, where a Lua function must be defined for each AST element type. Classic style writers are deprecated and should be replaced with new-style writers if possible.

  • Update https://github.com/jgm/pandoc/blob/main/doc/custom-writers.md#changes-in-pandoc-30 to link PANDOC_DOCUMENT / PANDOC_WRITER_OPTIONS to https://pandoc.org/lua-filters.html#global-variables
  • Update https://github.com/jgm/pandoc/blob/main/doc/custom-writers.md#example-modified-markdown-writer to link pandoc.write to https://pandoc.org/lua-filters.html#pandoc.write
  • Update https://github.com/jgm/pandoc/blob/main/doc/custom-writers.md#reducing-boilerplate-with-pandocscaffoldingwriter to link through to https://pandoc.org/lua-filters.html#module-pandoc.scaffolding / etc + the various other mentioned functions to their respective docs
• docs/customizing-pandoc.md has a lot of TODO sections currently.. could be improved, but is probably OOS of the main things being raised in this issue.
  • Add a manual (or manual section): Customizing Pandoc #3288
• MANUAL.txt
  • Update the djot-writer.lua links to point to relevant modern examples, and/or to just refer to doc/custom-writers.md or similar.
  • In general, ensure that the manual is aligned with what is said in custom readers/writers, and links to the more detailed documentation.
  • Probably also double check that it does similar for the lua filters documentation, etc
• Maybe other things, but thats all I have off the top of my head at the moment

[0xdevalias]

RE: AST Node Reference

I couldn't find a neat canonical list

We should have that.

This issue has a few mentions about it:

• Add a manual (or manual section): Customizing Pandoc #3288

  • filters (including documentation of the JSON serialization format and AST definition)

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • Definitely, documentation on pandoc's AST is something I've been longing for a long time.

    • But haven't managed to find a document that lists all of pandoc's AST supported elements.

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • per definition (pun intended), see jgm/pandoc-types@master/Text/Pandoc/Definition.hs#L95

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • When I was looking at the filter docs (related: #8750), I start wondering where I'd find the AST elements, and I started looking for the reference. That led me to this issue.

      I think I'm going to try to make it a hobby project to put together some AST docs. I can start by looking at the code in src/Text/Pandoc/Definition.hs>

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • Isn't Text.Pandoc.Definition's haddock page a good canonical reference for the AST elements?
      I'm curious what more you think would be needed in the way of documentation.

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • One of the downsides of the Haddock page is that is contains a lot of info and can be overwhelming. E.g., I'd guess that it's not immediately obvious to a user unfamiliar with Haskell that the list of instances can be skipped at first reading, but that the constructors are important: instances take up half my screen when I load that page, while constructors are just two lines.

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • I totally agree with @tarleb on the current docs being overwhelming to non-Haskell users.

    • Ideally, I'd love to see that pandoc would include a JSON or (YAML) file with the full AST specification (node name, type, attributes, etc.). If pandoc could auto-generate this JSON/YAML file (either within the source repository, or directly from the pandoc executable binary) and then provide a CLI command to emit it (e.g. --print-ast-spec) it would make life very easy, since it would be available without having to surf the web for that info.

  • Add a manual (or manual section): Customizing Pandoc #3288 (comment)

    • +1 on autogenerating. What I want to avoid is a manually produced document that could get out of sync.

[jgm]

I'd like to get @tarleb 's fedback on your comment, but to me these all seem like good changes! Thanks for the thorough look-through.

[rnwst]

Two thoughts on @0xdevalias' comments above:

• In this comment you noted that it would be nice to have default writer functions defined in pandoc.scaffolding.Writer, which can then be overwritten with custom behaviour. I also think that this might be desirable (note that this would require specifying an output format when initialising the Writer table). Currently, if you want to tweak the behaviour of an existing Writer, a Lua filter is your best bet. The filter would return a RawBlock or RawInline element of the specified output format for the elements whose output you want to modify. If you have a concrete use case and need help with this, please start a discussion.
• You commented that you were looking for a reference of AST elements. The official reference is Text.Pandoc.Definition (I think you worked this out already). This is referenced in the Filters documentation. Perhaps this should also be mentioned in the Writers documentation (and maybe in other places too).

You make some good suggestions for improving documentation in this comment, however I find it really hard to find the relevant information when your comments are so incredibly verbose. It would be easier to follow if you only included the most relevant information, or summarised your findings instead of listing them in great detail.

For anybody else coming across this issue, the relevant recommendations for improving documentation are in the second half of this comment, after the sentence

Based on this discovery, a few things that would probably be worth doing: