How are project and document yaml metadata merged, and how can they be accessed both in lua filters?

[ute]

Description

I am writing a quarto extension that makes use of some extension specific yaml list keys. It would be great to have the possibility to define parts of the list in the project _quarto.yml, and parts in the document itself. However, the filter can see the project wide list entries only if there are no further entries in the document yaml header. During compilation however, quarto writes all of them into the output, nicely merged.
Is there a trick to retrieve all of them from the metadata?

Here is a mwe:

_quarto.yml:

Ykeys:
  A: "aha" 

document merging_yaml.qmd:

---
Ykeys:
  B: "better"
filters: [ykeysout.lua]
---

## How are yamls merged?

Let us see in this document.

filter ykeysout.lua:

function Meta(meta)
  if meta.Ykeys 
   then
      print("found a Ykeys entry!")
      for k,v in pairs(meta.Ykeys) do
            print(k .. ": " .. pandoc.utils.stringify(v))
      end      
   end
   return(meta)
end   

Rendering this document in quarto 1.7.33, both entries A and B are shown in the output, but only B is visible to the filter:

pandoc 
  to: html
  output-file: merging_yaml.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  variables: {}
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  Ykeys:
    A: aha
    B: better
  
found a Ykeys entry! 
B: better

The only trick I could come up with is to use different key names for project, directory and document yaml, but I hope there is a better solution 😄

p.s. if the key is not set in the document, but only in the project yaml, the merging works as expected and the filter outputs the value from the project yaml. Also, if a _metadata.yml is present, and no Ykeys in document yaml, both keys from _quarto.yml and _metadata.yml appear in the Pandoc metadata as expected.

[mcanouil]

For reference, documentation on metadata is there:
https://quarto.org/docs/projects/quarto-projects.html

This being said, Last time I checked level project metadata are not exposed to Lua and there was issues:

• Expose project metadata to Lua filters #13029
• Some metadata set in _metadata.yml and in YAML header inside document are not correctly merged.  #9864

Without writing a Lua filter, you can have an idea of what is accessible using the meta shortcode: {{< meta Ykeys.A >}}.

[ute]

Thank you for replying so quickly :-) on a Sunday... and for the links.
I actually want to be able to access the correctly merged metadata in a lua filter, but this meta shortcode is nifty and shows the same problem immediately.

[mcanouil]

Current behaviour is document overrides completely Ykeys from project.

From the documentation "Document options (overrides directory and project)".

If you want to merge, you need to use _metadata.yml next to your document.

[ute]

Thanks again, @mcanouil!

Yes, _quarto.yml and _metadata.yml do merge nicely, but document yaml does not merge with these two. I will define an extra key, Ykeys-merge, say, for use in the document, and merge them within the extension.

"Current behaviour" sounds like you are having plans to streamline merging of yamls at some point in time :-). Quarto keeps getting more and more awesome 👍 and more and more complex.

[mcanouil]

"Current behaviour" sounds like you are having plans to streamline merging of yamls at some point in time :-). Quarto keeps getting more and more awesome 👍 and more and more complex.

Well, I can't speak about that but the team is aware of issues with how metadata are managed across document, configuration files, and CLI flags.