Does Quarto and Nextflow work together?

[oandrefonseca]

Description

Hi all,

I have been trying to use Quarto in a very peculiar way. I am using Quarto to produce reports in a Nextflow pipeline, i.e., every task is done either using R or Python through Quarto and then I get a task-specific HTML report. At this level, it is a pretty straightforward strategy, and works just fine :)

However, I would like to step further and create an entire website based on these reports. I am trying to figure out a nice way to do it, for instance, leveraging the freeze + caching from Quarto. Once I don't want to recompute things.

1. Has anyone faced something similar?
2. Additionally, I noticed that Quarto does not render a project if all its content is composed of symbolic links. Is there a workaround for it?

Any thoughts would be highly appreciated.

[mcanouil]

It really depends on what the components of your project will be.
For instance, changing the navigation bar or sidebar means to recompute several elements.
This being said, if you target the specific directory/file you want to render in a project, it's possible to do something more "incremental".

Overall, I don't see any constraints or using Quarto and Nextflow, but as with any workflow manager, you have to think about how you are going to make this happen.
I would suggest to simply try.

One of the main focus of Quarto is reproducibility. Symbolic links are basically against that idea.
Also I believe, this is a constraint from Deno (@cscheid could confirm this).

[cscheid]

Additionally, I noticed that Quarto does not render a project if all its content is composed of symbolic links

It's hard to get symbolic links to behave consistently across all operating systems, and so we don't generally support them.

If you're on UNIX and both projects are in the same filesystem, you could hard-link them. Quarto should be unable to tell the difference then.

Another possibility is to make a quarto project in the parent directory of your Nextflow project, and then configure your project to determine which files should be handled by quarto: https://quarto.org/docs/projects/quarto-projects.html#render-targets

[oandrefonseca]

@mcanouil

I have been playing around with this Nextflow + Quarto combo. But I couldn't make work it yet. In my mind, I thought it would be similar to generating several HTML and then performing multiple includes into a landing page.

So, imagine a pipeline that has three processes: PROCESS_A, PROCESS_B, and PROCESS_C. Each process generates a single report. I want to create a fourth report/HTML that integrates all reports. This being said, I thought I could render pieces individually and then migrate each HTML, QMD, and *_cache/files to a final quarto project (PROCESS_RENDER_ALL). I tried it using both the "cache" and the "_freeze" folder, but it did not work.

As for the sidebar, I thought I could leverage the Auto Generation feature from Quarto.

I am trying to figure out what I did wrong... Do you have any tips?

@cscheid

I could do a workaround on the symbolic links. Nextflow has stageInMode that allows us to copy data on the "work" directory rather than using symbolic (default behavior).

Sorry for the silly question. Thanks very much :)

[mcanouil]

I am not completely familiar with Nextflow (I basically know what it is and how it works, but never had the opportunity to use it).

Here the idea would be:

• add a Quarto document within a Quarto project (possibly a website)
• quarto render new-file.qmd with freeze option and possibly caching.
• quarto render at the project level to render common components such as sidebar, etc.

[oandrefonseca]

I think we have similar ideas. But let me disturb you a bit more :)

Based on your steps:

1. I will have a sort of "template" website ready on my pipeline repository, e.g., assets/template;
2. Every process will heritage that template + a .qmd related to that specific process/module (e.g., process_A.qmd, process_B.qmd, etc.);
3. During the execution process I will have to render that specific .qmd file (e.g., quarto render process_A.qmd);
4. As output my process will export the _freeze + all process_A* files, including the _cache and _files directories;
5. This process will be repeated for each process/module in my pipeline;
6. Finally I will migrate all outputs to a final module and then perform the project-level rendering, i.e., quarto render.

Thanks

[mcanouil]

Yes, something like that.
Although, I think for each new file, you only need _cache and _freeze. The rendered files can be obtained by the project level rendering.

[ran-codes]

did you get this to work?

(I think I am in the same boat. Want to use Quarto notebooks to capture individual pipelines then looking for an orchestrator for these individual pipelines either snakemake or nextflow.)