Constructing ShortCodes in Filters

[acederberg]

Description

Hi!

I have a bit of a tricky question. I wanted to inject a shortcode using a filter to have it rendered by another filter later on. My usecase is to create the shortcodes for iconify from some metadata. I tried injecting my shortcodes inline, e.g. adding it using pandoc.Str but this did not work.

Minimal Example

-- example.lua
-- This matches the div with id ``example`` and overwrites its content.

function Div(el)
  if el.attr.identifier == "example" then
    el.content = {
      pandoc.Header(2, "Test"),
      pandoc.Para("I Would Like an Icon Below"),
      pandoc.Para({
        pandoc.Str("{{<"),
        pandoc.Space(),
        pandoc.Str("iconify"),
        pandoc.Space(),
        pandoc.Str("devicon"),
        pandoc.Space(),
        pandoc.Str("python"),
        pandoc.Space(),
        pandoc.Str(">}}"),
      }),
    }
  end
  return el
end

return { Div = Div }

and

---
# example.qmd
title: An Example
filters:
  - example.lua
  - iconify
---

::: {#example}

This content is removed by the filter.

:::


## What I want:

The section above should render a python icon {{< iconify devicon python >}} instead of a shortcode.

Rendering this gets me

[mcanouil]

I'm not sure it's possible.
If it is, it's via the order of filter which is documented but a bit hidden unfortunately: https://quarto.org/docs/prerelease/1.4/lua_changes.html#more-precise-targeting-of-ast-processing-phases

[acederberg]

Temporary Fix

I imagine I could just insert iconify tags using pandoc.RawInline and that'll be a fine temporary solution. I noticed that if I put shortcodes in the metadata then I can get the icon that way using the iconify filter.

By the way, thank you for the iconify filter. It works great.

AST Analysis

I added a print statement to print out each pandoc.Para

function Para(el)
  print(el)
end

and recovered the folling:

Para [Str "The",
Space,
Str "section",
Space,
Str "above",
Space,
Str "should",
Space,
Str "render",
Space,
Str "a",
Space,
Str "python",
Space,
Str "icon",
Space,
Span (
  "",
  [],
  [
    ("__quarto_custom","true"),
    ("__quarto_custom_type","Shortcode"),
    ("__quarto_custom_context","Inline"),
    ("__quarto_custom_id","1")
  ]
) [],
Space,
Str "instead",
Space,Str "of",Space,Str "a",Space,Str "shortcode."]

So it might be possible with spans. Will update if I can make it happen.

I see that shortcodes are parsed in a typescript file.

[cscheid]

quarto.Shortcode {} is supposed to work, but there's a bug there. Hold off just for a bit and we'll get this sorted out.

[acederberg]

Cool! Thank you.

There are still a few things that are unclear that I wish to know that I am having some trouble with.

My (Actual) Usecase

If I can make things a little worse, I need to know how to make the pandoc.Span for subsequent processing since I am not using a lua filter in my actual usecase (I am using panflute for a json filter).

How to Construct Shortcode Equicalents using pandoc.Span?

I looked around and am unclear if I understand what quarto does with the spans. It looks like the spans are parsed by this typescript file and then handled by this lua file. But I cannot tell how to add parameters for the shortcode function to the pandoc.Span.

[cscheid]

Unfortunately, if you need to do this in a JSON filter, the answer is currently "you can't".

Let me try to fix it and put an example document together.

[cscheid]

Ok, I have a PR up that enables this, together with a minimal JSON filter that adds a new meta shortcode to the document: #11241

As you can see by the Python file that represents the JSON filter, this is quite tricky. We'll need to document the usage properly, but you might be able to simply copy and paste bits of https://github.com/quarto-dev/quarto-cli/blob/0e467b60a303c085faad877914d35ca52453f7df/tests/docs/smoke-all/2024/10/29/test-shortcode-json-filter-insertion/insert_shortcode.py to your needs.

A JSON filter that wants to interact with custom AST nodes needs to know:

How to encode the custom data accessible by the custom AST node

In JSON filters, Quarto emits a list of dictionaries in the quarto-custom-node-data metadata entry of the Meta field in the Pandoc object. Each custom AST node uses a different table, but the important common bit of information they all hold is the "id" of the custom node, which allows Quarto to associate this custom data with the Pandoc element it represents. That is the quarto_custom_meta field in each of these objects (note the _ vs - distinction here compare the metadata key itself). I'd recommend you refer to the example JSON filter, as well as writing one of the JSON inputs toa file and navigating it with your favorite JSON editor/browser.

How to encode the node itself in the Pandoc document

Each custom AST node is represented by either a Div or a Span in the document with special attributes. Again, the easiest way for you to get started is to refer to the example filter in the PR I shared.

The overall process to create a new shortcode invocation, then is to:

• add a new node to the main blocks field of the Pandoc document in JSON, and decorate this node with the correct classes and attributes, including a unique __quarto_custom_id id that needs to be the string representation of a unique integer starting from 1.
• add a new table to the quarto-custom-node-data entry of the Meta field in the Pandoc document, whose "quarto_custom_meta" field contains an object with itself a field "id" that matches the __quarto_custom_id attribute above. This table also needs the rest of the custom node information (for your case, refer to the example document)

(If you're reading this and are attempting to create more complicated nodes like Callouts or FloatRefTargets from a JSON filter, know that I haven't shared all the details here; we'll work on this in the coming weeks)

[cscheid]

@acederberg If the tests pass, I'll merge this into our 1.6 prereleases which should get you unblocked.

[acederberg]

Thank you so much! I really appreciate it.
I'll let you know once I have tried this functionality out.