How to parse a qmd file in Python?

[kcarnold]

Description

I'd like to parse out the headings of all of the qmd files in a repo, to make a sitewide table-of-contents for example. (Is there any existing thing that does that?)

So, grab all of the lines that start with # in all the .qmd files, right? Not so fast; most #-lines are code chunk comments.

Before I go writing a parser and/or chatting with pandoc, has anyone else done this already?

[mcanouil]

That's what Pandoc/Quarto does, .i.e parsing/reading markdown.
You can use LUA filter or any script as a post-render script.

[mcanouil]

I was not talking about using a Python or third party script, but to use Quarto/Pandoc directly with a LUA filter to extract headings.

[cscheid]

(sorry for the late response)

Learning our Lua API and using it really will be your best bet.

You could try to use panflute (a Python library that helps write pandoc JSON filters in Python), but Panflute doesn't recognize a number of Quarto-specific markdown features (shortcodes, FloatRefTargets, callouts), and so it will be a little harder than usual.

I'll also note that going through pure Pandoc will not be a great fit, because Quarto uses a custom reader, and so our Markdown syntax is not precisely that of Pandoc.

[kcarnold]

Thanks. For my specific need, the Python script I posted above, using panflute, worked well enough. But for future readers of this thread, could you give an example of what that script would look like using Quarto's custom reader?

In particular, all of the examples on the website are Quarto calling a Lua script as part of a normal rendering process. What if you wanted to use the Lua API from outside of that context (e.g., another script doing a different task)? Would you just need to pretend you're rendering something, have Lua write to a file or something, and then throw away the render output?

[cscheid]

(GitHub's post-hiding features got me here, and I completely missed your panflute code, sorry!)

But for future readers of this thread, could you give an example of what that script would look like using Quarto's custom reader?

Our filter infrastructure is not particularly well suited for the task of walking the entire project. I think this is a use case where your script is actually the best strategy. I'd love to figure out a way to help, but I'm going to have to think about this a little bit.

What if you wanted to use the Lua API from outside of that context (e.g., another script doing a different task)?

You can do quarto run file.lua, which, under the hood, does exactly the dance you're suggesting, and you'll have access to the Quarto APIs: https://quarto.org/docs/projects/scripts.html#periodic-scripts

Unfortunately, in this case there won't be an easy way to get to all of the .qmd files, open each of them, and so on.

Although I think this is a niche use case, it's one where "when you need, you really need it". I'm going to open a new GitHub issue to mull over this.

Sorry I missed the original discussion (a confluence of multiple travel and covid had me offline for almost a month), this is a pretty important use case that we want to support!

[cscheid]

Tracking on #10539.