How can I determine which files will be rendered within a pre-render-script's execution during incremental directory-rendering?

[Gewerd-Strauss]

Description

Hello,

Assume I have a project with a pre-render script pre-render-1.R, e.g. the following principal structure:

.quarto
	...
experiments
	___about.md
scripts
	pre-render
		pre-render-1.R
___about.md
index.md
_quarto.yml

Assume execute-dir: project for this project. This is relevant because files from different directories must access things in various subdirectories, which are usually not relative to the document. Doing that manually will be an absolute nuisance.

Further assume that pre-render-1.R must perform some lengthy operation (10-20 seconds per file) on every .md-file in the project*. This step cannot be skipped, and cannot be cached. At the moment, the following method is used in the pre-render script:

a <- quarto::quarto_inspect()
root <- a$dir # get the project root.
...
pattern <- paste0("\\___.*\\.md")
files <- list.files(
    path = root,
    pattern = pattern,
    full.names = TRUE,
    recursive = TRUE
)
for (file in files) {
    # some time-intensive operation
}

Quarto allows incremental and project-wide renders, via quarto render <directory> and quarto render respectively, executed within the project directory.

During the R-script's execution, the relevant files to operate upon must be collected. Currently I retrieve the project's root as shown above, then parse the projects' contents and go from there.

That is obviously fine for project-wide renders.

For directory-restricted renders, I would love to use that path to reduce the number of files pre-render-1.R will execute upon. If I want to only render /experiments/, I would like to have this script only search through that subdirectory, and only operate upon /experiments/___about.md.

Unfortunately, quarto::quarto_inspect() does not return this information. I am currently not aware of any method of retrieving this info for use in a script that would allows this. As far as I know, this is simply not possible.

Is this correct?
If so, could this be implemented?

---

Thank you.
Sincerely,
~Gw

---

* Simplification. Only on files with a certain prefix in their name get processed.

---

P.S.:
I was intending to open this discussion on the quarto-r-repo. As discussions are not available there, I opened it here instead.

[mcanouil]

I don't understand your question/setup.

"pre-render" script is a Quarto feature but it seems you are talking about something else, see https://quarto.org/docs/projects/scripts.html#pre-and-post-render.
It seems you reimplemented this feature in R, then call Quarto from R, etc.
It would probably be simpler to use Quarto's features directly and set an R script as the pre-render which can leverage Quarto environment as documented in the link I shared.

FWIW, even if I understand the appeal to do everything from R, here it seems a bit cumbersome and more complicated than providing an R script to Quarto and simply render the project.
You probably have some other constraints that explain this design choice.

To be clear, the R package is only a wrapper around the real tool which is Quarto CLI.

[Gewerd-Strauss]

I might have explained badly.

It seems you reimplemented this feature in R, then call Quarto from R, etc.

&

providing an R script to Quarto and simply render the project.

I am aware that pre-render is a quarto-feature, and I am not re-implementing it.

I run quarto from cmd.exe normally via quarto run <target>, which triggers the set of pre-render scripts. These scripts are written in R.
One of these scripts calls an external executable via processx::run() as follows:

a <- quarto::quarto_inspect()
root <- a$dir
app <- normalizePath(str_c(root,"/scripts/lib/app.exe"))
pattern <- paste0("\\___.*\\.md")
    files <- list.files(
        path = root,
        pattern = pattern,
        full.names = TRUE,
        recursive = TRUE
    )
args_ <- c("a=b","c=1","d=-1","--noMove")
for (file in files) {
        if (str_count(file,"_vault.md")>0) {
            next
        }
        args <- args_
        args <- c(args,str_c("path=",file))
        run(app,args,echo = T)
    }

Executing this script takes time; however I cannot skip this step for reasons which are complicated and intertwined with my setup. It can be assumed that prior to rendering, this script must call its executable on these files.

If I run a partial render of /experiments/, currently the code above will execute app for all files in the project which satisfy the pattern:

pattern <- paste0("\\___.*\\.md")

That is, it will execute over

___about.md
experiments
	___about.md

instead of only running over

experiments
	___about.md

So I want to be able to detect - during the execution of this script as part of the pre-render pipeline issued by quarto render [<directory>|<path/to/file>] - which folder is being rendered.

Is this a project-wide render? Then execute app over all files satisfying pattern.
Is this a subfolder-only render? Then only execute over those matches.

However, the quarto-r-wrapper does not seem to expose this difference.
For both a project-wide-render and a single-file-render quarto::quarto_inspect() will return the same results.¹

In principal even just exposing the raw input-string handed to quarto.exe (i.e. quarto render /experiments/, quarto render, quarto render path/to/file.qmd --no-clean --quiet) would allow this. Or any other way of retrieving this info during the execution of a pre-render script.

I hope this explains it a bit better?

~Gw

---

Footnotes

1. I am not even sure if that should be quarto::quarto_inspect()'s task to begin with, it doesn't really fit. ↩

[mcanouil]

I'm confused why don't you use the environment variable as documented in the pre/post render documentation if you are using this feature? https://quarto.org/docs/projects/scripts.html#pre-and-post-render

• QUARTO_PROJECT_INPUT_FILES

  Newline separated list of all input files being rendered (passed only to pre-render)

[Gewerd-Strauss]

Huh.

I am dumb.
This isn't exactly what I need (I'd need the target directory itself), but I can work with this.
Because the input-files I look for in this script begin with underscores, they will not show up in this data. This is by design.

So I will just parse the root of the paths given, check if they are all the same, and then I can run list.files() over that root.

Thank you.

[mcanouil]

I would suggest to use the R package fs for this task to identify the root, etc.
For instance, it has a path_has_parent() function.