Fetching Qmd metadata from code to compile a list of articles?

[capnrefsmmat]

I'm using the Quarto website format to build a repository of datasets for statistics courses. So each dataset has a Qmd file describing it, and I'm using the Listings feature to make a sortable page listing them all.

I'd like to add custom metadata to each Qmd, such as the year the data was collected, the number of variables, or the license of the data. I can display some of that in the listing, but it'd also be very useful to compile all that data into some kind of spreadsheet, so course instructors could quickly find datasets that meet their requirements (and so Quarto doesn't have to reinvent searching and sorting tabular data).

In principle this should be easy, since it's just YAML metadata in the headers of each Qmd. Is there a standard API for fetching the metadata of all articles that I could use from R or Lua? Then I could build the tool to export a spreadsheet easily. Failing that, is there a reason it wouldn't work to simply read all the Qmd files and their header YAML from a script, and use that to generate a CSV?

[mcanouil]

You could do the other way around, write the spreadsheet and use it to fill/create the listing, see for example how I use "pre-render" script to build my publication listing out of a bib file: https://github.com/mcanouil/mickael.canouil.fr.

[capnrefsmmat]

Interesting. So you have a source file (in your case, a .bib file) and an R script that converts it to YAML that the listings feature understands?

I wanted to include the metadata in each Qmd so I could also put it into the article template, so it displays with the author, date, and other metadata. But maybe there's another way to get the metadata into the data description pages.

[mcanouil]

You could also read/parse all the Quarto documents in a folder, for example in R, there is the R package yaml.

[andrewpbray]

Sounds like a great project!

I've been bumping into similar questions and I find there are two uses cases: how to centralize content from many qmds into a single file, and how to distribute content from one file to many qmds. It sounds like you're firmly in the centralizing use case.

Just spitballing, but could you:

1. Add an R cell to the page where you'd like users to be able to search the data sets.
2. In the cell, map rmarkdown::metadata across the list of qmd files and gather the results into a data frame.
3. Render the data frame as a DT data table, which would be searchable and filterable (and you could embed links to the qmd landing pages for each data set).

[capnrefsmmat]

That sounds like a pretty nice idea. If DT supports all the searching and filtering we might want to do, this would solve most of the problem. I'd just have to choose a schema for the metadata I want to include in each dataset and write some R to extract it.

I'm going this route because I'd like each qmd to be self-contained. I figure that's probably best for long-term maintenance -- in projects like this, hosted in an academic department and developed in spare time, any system that requires complicated coordination is going to eventually fall apart. And 10 years from now there's something new past Quarto, the Qmds are just Markdown with some metadata at the top, and it wouldn't be too hard to convert that into whatever new system we need.

[cscheid]

In principle this should be easy, since it's just YAML metadata in the headers of each Qmd. Is there a standard API for fetching the metadata of all articles that I could use from R or Lua?

Currently there isn't a clean API, but this is a totally reasonable request! I'm going to mark this for our quarto 1.5, where we're considering a revamp of project-wide rendering. Quarto will collect and reason much more explicitly about inputs and outputs, and providing a project-wide tabulation of document metadata is a natural thing in that context. I can't promise that we'll be able to do it, but it seems like it could have a number of other useful applications.

In the meantime, if you want to collect this information as your website renders, you can add a Lua filter that operates on the Meta entry, and writes the metadata to a JSON file somewhere. It would be your responsibility to operate on this JSON file as necessary later on. Here's an example project I just put together that shows the technique. If this isn't sufficiently self-explanatory, please let me know!

[capnrefsmmat]

Interesting, I'm testing that out now. Combined with @andrewpbray's suggestion of using DT to make a filterable table, I can get it to make a nice table.

The only problem is order of operations. If the page containing a table of metadata is rendered as page 5 of 10, only four metadata entries are stored in metadata.json, and the rest have yet to be generated. Is there a way I can force it to be rendered last?

I suppose the alternative would be to modify the pre-render and post-render scripts so the previous metadata file is kept unchanged during rendering, and then replace it with the new metadata once rendering is complete. Then the table would include only the metadata that's been previously calculated, not any changes in the current render.

[capnrefsmmat]

Alternately, I could go back to generating a downloadable CSV at post-render, rather than making an interactive table. That would also be fairly easy, and I'd keep using ordinary listings for browsing on the website.

[cscheid]

The only problem is order of operations. If the page containing a table of metadata is rendered as page 5 of 10, only four metadata entries are stored in metadata.json, and the rest have yet to be generated. Is there a way I can force it to be rendered last?

Not yet, unfortunately. This kind of fine-grained dependency control is exactly the sort of feature in scope for our planned changes in 1.5, though.

[loneguardian]

I would like to point out that the Dataview plug-in for Obsidian is a great example of this functionality. In Obsidian, one can easily perform queries on metadata using code blocks.

I am looking forward to having such function to run a frontmatter powered CMS for many things, e.g.:

• to track status for each file, something like an issue tracker, or progress monitoring
• to curate article collections