Can I update only what changed between previews

[statquant]

Hello,
We are using rmarkdown to create reports with a lot of dynamic graphs (plotly-like). As it appeared that rmarkdown was too slow to generate the reports and the reports were too slow to be rendered in the browser we came up with a solution that would save an image of the graph and a clickable link to the standalone dynamic plot (which is saved as html on another non blocking thread).
With this we're back to a workable state.

Something we never really managed to do is to be able to use previews, we know about caching but it means loading the cache rds file and that's too slow. We improved the format of the cache to use qs as opposed to rds but the problem remains.

Ideally we'd like to be able to use preview by rendering the document again and again in the same session (so data stays available) but only regenerating the part of the markdown document that correspond to a chunk or cell that changed since last save. Said otherwise if a chunk has not changed it's like it would be eval=FALSE (but the output in the markdown output would stay).

Is it something that can be achieved with quarto (we do not care switching from rmarkdown to quarto and from say knitr to jupyter for the engine)

[sasja-san]

Hello!

Did you forget something? ;)

[cscheid]

To do it right, this is a feature that's unfortunately much harder than it looks. With that said, if your code cells are the source of most of the computation, maybe you can use something like the knitr cache?

[statquant]

I do not think I can do anything with caching, we are working on fairly big data, loading the processed (cached) data is simply too slow.

I did something ugly, I split chunks into data-processing chunks and plot chunks and I created a hook that allows to not eval the chunk if a (list of) objects already exist in the session.
This way I can manually re-render and skip most of the work. But... I still have to render the plots, that does not work in preview (because if I'm right preview re-knit from scratch) and it requires to reference the hook on selected chunks (too constraining for the user)

[mcanouil]

Using targets and parquet files could help improving performance.
RDS files are not the only way to store objects.
You can also disable compression which will improve loading times.

[statquant]

Thank you, as I said above we replaced rds files by qs which can be decoded on several threads and although it's better it's still the same issue for a long Rmd file with 10s of chunks you spend minutes loading cache that you should not really be loading (you do not load cache fro cells in a jupyter notebook when you execute the last cell).

I agree targets is likely to be a solution but learning targets and the constraints it put on the structure of the Rmd is not something people want to pay.

I don't mean to be overly negative, I was wondering if out pb was solved somehow with quarto that's all.