header-includes: metadata and defaults file inconsistencies

[periodicpoint]

Context

I am transpiling form Markdown to PDF via XeLaTeX.

Pandoc version

pandoc 3.1.11.1
Features: +server +lua
Scripting engine: Lua 5.4

OS

Arch Linux
TeXLive: texlive-full Version: 1:2024.20240331-1

Goal

I want to add additional LaTeX packages ad hoc without changing the existing template.latex.

Confusion

Case 1 (expected to work)

If I include the extra LaTeX package via header-includes: in my metadata file and link it from my defaults file, everything works as expected:

File: metadata.yaml

header-includes:
 - '\usepackage{libertinus}'

Case 2 (expected to not work)

If I include the extra LaTeX package via header-includes: in my defaults file it is not working, which is at least somewhat expected (although the docs are not clear in this case):

File: defaults.yaml

header-includes:
 - '\usepackage{libertinus}'

Case 3 (expected to work)

If I add the extra LaTeX package via include-in-header: in my defaults file it is not working, which is totally unexpected.

include-in-header:
  - 'inc.tex' # Containing the line \usepackage{libertinus}

This is unexpected in particular because the docs say:

header-includes
contents specified by -H/--include-in-header (may have multiple values)
(https://pandoc.org/MANUAL.html#variables-set-automatically)

What the docs say

Doc A

Raw content to include in the document’s header may be specified using header-includes; however, it is important to mark up this content as raw code for a particular output format, using the raw_attribute extension, or it will be interpreted as Markdown. For example:

header-includes:
- |
  ```{=latex}
  \let\oldsection\section
  \renewcommand{\section}[1]{\clearpage\oldsection{#1}}

(https://pandoc.org/MANUAL.html#extension-yaml_metadata_block)

Doc B

To override or extend some CSS for just one document, include for example:

---
header-includes: |
  <style>
  blockquote {
    font-style: italic;
  }
  tr.even {
    background-color: #f0f0f0;
  }
  td, th {
    padding: 0.5em 2em 0.5em 0.5em;
  }
  tbody {
    border-bottom: none;
  }
  </style>
---

(https://pandoc.org/MANUAL.html#variables-for-html)

Doc C

make \paragraph and \subparagraph (fourth- and fifth-level headings, or fifth- and sixth-level with book classes) free-standing rather than run-in; requires further formatting to distinguish from \subsubsection (third- or fourth-level headings). Instead of using this option, KOMA-Script can adjust headings more extensively:

    ---
    documentclass: scrartcl
    header-includes: |
      \RedeclareSectionCommand[
        beforeskip=-10pt plus -2pt minus -1pt,
        afterskip=1sp plus -1sp minus 1sp,
        font=\normalfont\itshape]{paragraph}
      \RedeclareSectionCommand[
        beforeskip=-10pt plus -2pt minus -1pt,
        afterskip=1sp plus -1sp minus 1sp,
        font=\normalfont\scshape,
        indent=0pt]{subparagraph}
    ...

(https://pandoc.org/MANUAL.html#layout)

Doc D

include-before-body:
  - inc.tex

(https://pandoc.org/MANUAL.html#general-writer-options-1)

Questions

1. In doc case A it seems that we need raw code fencing, yet in doc case B and C the manual seems to suggest that fencing is not needed. Why is this not consistent?
2. Doc case A-C seem to cover metadata variables that should go into metadata files. In case of D there seems to be another general option to include additional TeX content via include-before-body:. Yet, including extra latex packages via this variable does not work, because there seems to be no such variable in the template.latex file. (The header-includes: is present in the template file.) Why is this so?
3. Why are extra LaTeX files handled via the metadata file, while they are core components of the document and most definitely not metadata?
4. The docs say:

> header-includes
>     contents specified by -H/--include-in-header (may have multiple values) 
> include-before
>     contents specified by -B/--include-before-body (may have multiple values) 
> include-after
>     contents specified by -A/--include-after-body (may have multiple values)

(https://pandoc.org/MANUAL.html#variables-set-automatically)
Why then, does the latex template not simply contain the corresponding variables header-includes, include-before and include-after directly instead of header-includes, include-before and include-after?

Potentially related question

#10361