Default path for filters?

[njbart]

I’m not sure I have been looking in the right places, but I wasn’t able to find any documentation on default paths, in particular for (pandoc) filters.

Based on the behaviour of Rmarkdown, I was expecting to be able to put a filter in a filters subdirectory of pandoc’s “User data directory”, typically (at least on macOS) /Users/<username>/.local/share/pandoc, which would be found upon rendering without giving the full path.

quarto pandoc -v output does indeed include User data directory: /Users/nb/.local/share/pandoc

Now, my filters live in /Users/nb/.local/share/pandoc/filters, but when using a file containing

filters:
  - helloworld.lua

rendering fails with

Error running filter helloworld.lua:
cannot open helloworld.lua: No such file or directory

whereas providing the full path /Users/nb/.local/share/pandoc/filters/helloworld.lua works as expected.

Is this a bug? And if not, what is the default path where quarto is looking for filters, templates etc.?

[dragonstyle]

Filter paths should be relative to the file that is referencing them (e.g. if you put it in document front matter, it will be relative to that file). So in the case of - helloworld.lua, we'll be expecting the filter right next to the input document.

[cderv]

Just chiming in that as I wondered the same and it was not all true.

Let's say I have this structure:

> tree
C:\USERS\CHRIS\APPDATA\LOCAL\TEMP\TEST-LIGHTBOX
└───lightbox
    ├───example_files
    │   └───mediabag
    ├───images
    └───_extensions
        └───quarto-ext
            └───lightbox
                └───resources
                    ├───css
                    └───js

And I am place in the root test-lightbox repo

> pwd

Path
----
C:\Users\chris\AppData\Local\Temp\test-lightbox

Should this work as _extensions is correctly at the level of the document ?

> quarto render .\lightbox\example.qmd --to html
pandoc
  to: html
  output-file: example.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Example Lightbox Document
  lightbox: auto

Error running filter lightbox:
Could not find executable lightbox

If I enter the folder of the .qmd then it works.

> cd .\lightbox\
> quarto render example.qmd --to html
pandoc
  to: html
  output-file: example.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Example Lightbox Document
  lightbox: auto

Output created: example.html

Trying to check we don't have a regression here.

[dragonstyle]

Thank you! Yeah this seems wrong (like we're using the working directory rather than the input directory when resolving the path to the extension). Extensions are a special case since there you are referencing an extension name rather than a filter path, but this should definitely work. An issue would be appreciated because this should be fixed!

[cderv]

Ok. I tested without an extension, and it is working. Issue opened #2351 for extensions

[njbart]

Thank you, but that sounds a bit inconvenient, in particular for filters that are used on a regular basis, e.g., pandoc-zotxt.lua, as discussed a very short while ago here. Wouldn’t it be possible to have quarto search pandoc’s “User data directory” as well?

[dragonstyle]

Perhaps - you could also consider using a _quarto.yml project file to attach the filter to all the files in the project or even a _metadata.yml file to attach the filter to files within a directory. In those cases the path will also be relative to the file that is referencing them. Would that work?

[njbart]

All valid points, but my typical use cases are standalone files in different directories, practically all of which require a bibliography, hence the repeated need for certain filters such as pandoc-zotxt.lua. Using Rmarkdown, the filter’s file name was all that was ever required. As to quarto, rather than introducing the overhead of having to create additional files such as _quarto.yml, I’d rather keep using full paths to filters then; I’d consider that just a very minor inconvenience after all. But maybe there is a chance of quarto being modified to look in pandoc’s “User data directory” as well if a filter cannot be found in the local directory?

[iandol]

I would also like the search fallback to check the pandoc user data folder. I keep my bibliography and CSL files there and would like to keep a consistent location between pandoc and quarto...

@njbart — what happens if you explicitly add --data-dir=~/.local/share/pandoc to the quarto command? Does Quarto use this or how about --resource-path=?

[njbart]

@iandol – No, neither --data-dir=~/.local/share/pandoc nor --resource-path=~/.local/share/pandoc seem to make a difference. Also, I was looking primarily at rendering files from within RStudio, where it doesn’t seem to be possible to set additional command line options.

In addition – while I still might be misunderstanding how quarto is supposed to be handling these issues in general – it seems that .csl files are not searched for in standard locations either (in this case, the csl subdirectory of pandoc’s user data directory): Neither csl: apa.csl nor csl: ${HOME}/.local/share/pandoc/csl/apa.csl in a YAML metadata header work, only the full path from root, csl: /Users/nb/.local/share/pandoc/csl/apa.csl does. What’s more, if there’s a symlink involved, even the latter path does not work any more.

I am not sure where quarto does search for .csl files, but here, too, I feel it would be very useful it it searched in pandoc’s default location for such files at least as a fallback.

I suspect that when looking at templates and other stuff that usually lives in pandoc’s user data directory (which I haven’t tried so far at all), similar issues will crop up, too.

[cderv]

Just reacting on this:

Also, I was looking primarily at rendering files from within RStudio, where it doesn’t seem to be possible to set additional command line options.

Quarto uses default files YAML structure to pass command line. So --resource-path should be resource-path. You'll find those in the reference page (Ex: https://quarto.org/docs/reference/formats/html.html#rendering).

However, data-dir is not among them because I believe Quarto use that already. I believe this is part of the limitation encountered in this discussion

quarto-cli/src/command/render/pandoc.ts

Lines 673 to 681 in 20d3265

	// We always use our own pandoc data-dir, so tear off the user
	// data-dir and use ours.
	const dataDirArgs = new Map<string, boolean>();
	dataDirArgs.set("--data-dir", true);
	pandocArgs = removePandocArgs(
	pandocArgs,
	dataDirArgs,
	);
	pandocArgs.push("--data-dir", resourcePath("pandoc/datadir"));

[iandol]

Thanks @cderv — fair enough it seems the design is to isolate quarto from a pandoc install concerning --data-dir

I tried to use resource-path to define the search for my bibliography and csl files

resource-path: /Users/ian/.local/share/pandoc
bibliography: Core.json
csl: csl/apa-ian.csl

but I get this error:

Aeson exception:
Error in $: parsing [] failed, expected Array, but encountered String

This method works from the pandoc command line: pandoc --citeproc --resource-path=/Users/ian/.local/share/pandoc/ --bibliography=Core.json --csl=csl/apa-ian.csl

I know that pandoc does not currently allow resource-path as metadata: jgm/pandoc#8262 (comment) and perhaps this is the problem; I have no idea how quarto runs pandoc...

[cderv]

I know that pandoc does not currently allow resource-path as metadata: jgm/pandoc#8262 (comment) and perhaps this is the problem; I have no idea how quarto runs pandoc...

YAML metadata in Quarto are a bit different than pandoc in the way that you can pass what is valid in a Pandoc's default file. And resource-path is there (https://pandoc.org/MANUAL.html#defaults-files)

I am not sure about the issue... Look like a bug 🤔

[njbart]

Ok, so just to understand a little better how quarto is supposed to work: Is there a quarto default directory for "csl" files, and if so, where?

[njbart]

Actually, my last question was meant quite seriously. If there is a default directory for csl files, I haven’t been able to identify it so far, and if there isn’t, what are the alternatives, and what would the quarto developers recommend using instead?

• Using an absolute path to the .csl file? (Possible, but tedious, visual clutter in the metadata block)
• Using a URL to indicate the .csl file? (Possible, but again tedious, visual clutter in the metadata block, requires network access)
• Copying one or more relevant .csl files into every single working directory? (Possible, but even more tedious, also takes away the possibility of automatic updates of .csl files)
• Using a _quarto.yml or _metadata.yml file, as suggested above? (Possible, but excessive for my typical use case, i.e., short reports where otherwise there wouldn’t have been a need for anything but the .Rmd file itself.)

Honestly, I don't find any of these alternatives convincing. So if there really is no default directory for csl files at the moment, you should by all means consider adding this feature.

[jjallaire]

Our biggest concern about the use of the data-dir is that it undermines reproducibility (all of the source code required to successfully render the project or document is not available in the directory). This is why we don't use Pandoc's data-dir (as not only are there reproducibility problems but there are also mysterious "bugs" which show up when various options and templates are injected from there.

I think the best way to manage this if you absolutely want a shared asset is to create a symlink from your document or project directory.

Note also that _quarto.yml will allow you to inherit a CSL definition across all files in the directory (including subdirectories).

[jjallaire]

To answer in a bit more detail, you can think of Quarto projects as something like a scoped, explicit version of the data-dir. With Quarto, you put all of that shared configuration/content in a _quarto.yml file and then just make sure the files you want affected by it are below it in the filesystem hierarchy. For example:

reports/
  _quarto.yml
  my.csl
  phase1
      report1.qmd
      report2.qmd

Where _quarto.yml has only this:

csl: my.csl

Or if you like a reference to a CSL file at an absolute path for sharing system-wide.

While this does create a bit more maintenance at least now you can transplant that filesystem to another machine and everything will work correctly (rather than depending on something in a data-dir that might be absent).

[njbart]

Many thanks for the clarification and detailed explanations.

While I do understand your motivation a little better now, I’m afraid I’m still not entirely convinced.

I tend to see csl files (and, to a similar extent, pandoc templates and pandoc filters) as something that only ever needs to exist in one location of a system or account, just like binaries, LaTeX .sty files, or, for that matter, quarto’s html themes – and not usually as something that’s project-specific.

Reconciling our apparently rather different viewpoints does not appear easy, but what about permitting a ~/.quarto or ~/.local/share/quarto directory which wouldn’t be created by default, but would, if created by a user, indicate they want this directory to be used as the user default directory. This could still be symlinked to its pandoc equivalent or, just as my own ~/.local/share/pandoc/csl and ~/.csl are merely symlinks to ~/Zotero/styles (which takes advantage of being able to install csl files from within Zotero, and have them auto-updated) a symlink to the latter.

Anyway, for now, I’ll probably resort to using full paths to csl files.

However, one question concerning paths in this context:
csl: /Users/nb/.csl/chicago-fullnote-bibliography.csl works as expected
whereas
csl: ${HOME}/.csl/chicago-fullnote-bibliography.csl fails with
File ${HOME}/.csl/chicago-fullnote-bibliography.csl not found in resource path.
Is this expected behaviour, or would it merit a bug report?