Will Quarto scale well with a large number of files?

[neeldhara]

Description

TLDR: I am using Quarto in a way that might scale up to (tens of?) thousands of tiny-ish .qmd files in the project. Will this be fine in the long run? I only render files that change, is that alright to keep doing or should I be rendering the whole project every so often?

---

As I mentioned elsewhere, I am currently experimenting with using Quarto as a PKM system for my personal notes. After a couple of weeks of use, I am up 500+ notes (this is not surprising since notes are atomic, tend to be small-ish - many of them are timelogging entries, for example). If I render the whole project, it's a good five minutes for the whole site to be generated.

However, my typical use case is that I am working on some fairly focused set of subdirectories - so I make some changes, and just render the main index.qmd file, and this only seems to regenerate the files that need regenerating, takes under a minute (usually instantaneous, but longer if there is some LaTeX involved).

There are some listings that sometimes don't get updated when I do it this way: for instance when I add a note under the people directory, the YAML header contains information about projects they are a part of (via PIDs), and the project pages have a listing of people involved in them: sometimes this does not update automatically when I add a new person into the system). When this happens I can just redo those specific pages manually.

Is this a reasonably sustainable way of using Quarto, or should I be generating the whole project every so often to keep things clean? I am really hopeful that I can continue using it with re-rendering only the delta-s... that way it feels like I can continue to use the system as it grows into thousands of files or a lot more.

A more specific question is this: I use listings a lot - for example, Quarto is essentially a reference manager for me now: I have a bunch of .bib files (I expect this number to be in the hundreds/low-thousands eventually) that pandoc converts to YAML, and I show a list of papers, and it's really helpful that I can filter by title, category, etc. on the front end. When I add new papers, will Quarto slow down in generating the listings? Does it have to rebuild them from scratch?

So far, IME, Quarto has been very fast in generating journal pages from ~hundreds of (tiny) timelog entries. This gives me hope for the other listing types... am I missing anything here?

On a somewhat related note, are there any pointers to Quarto being stress-tested with large-ish amounts of data? Something in the spirit of a Moby Dick workout?

PS. I am a somewhat literate end-user of Quarto but I don't really have a sense of how the internals work, so sorry if these questions are very naive and have obvious answers one way or the other!

PPS. Other than anxiety about future-proof-ness and scale, I am really enjoying using Quarto for my personal note keeping. It's awesome!

[neeldhara]

PS. Quick follow up - as a casual benchmarking attempt, I tried copying the text from the markdown file available on the Moby Dick workout page, and Quarto renders it just fine in under a minute or so, and subsequent renders --- with no further changes to the file --- take no extra time (as I think one would expect?)...

[mcanouil]

FYI:

• Slow start of preview/render #6751

[neeldhara]

Oooh, nice, 250+ gigs is a more than fair upper bound for the amount of material that I'll directly generate with .md files in my lifetime, I think, so this gives me plenty of hope! I just hope I'm good with never rendering the whole project, and just the individual files as I'm used to doing now.

For the record, I am using this as a digital inventory of sorts for stuff like videos (that are large files on their own right), but I keep those off the repo and point to them using Hookmark, which is working quite reliably so far. So most of the total folder size comes from the text files, and I'm at ~30MB with a couple of weeks of use, so even with some non-linear scaling, I don't see myself crossing 30-50GB. 👀

[cscheid]

PS. I am a somewhat literate end-user of Quarto but I don't really have a sense of how the internals work, so sorry if these questions are very naive and have obvious answers one way or the other!

We're sorely missing an internals guide to quarto, we apologize. We'll fix this in the next few months. Thanks for the patience!

[neeldhara]

Thank you so much! A sneak peek into Quarto internals would be lovely to have whenever it's feasible for it to be put together - I'd definitely look forward to reading it as a curious outsider. If there's anything I can help with, please let me know.

PS. To be clear I didn't mean to complain about the lack of a guide to internals. The Quarto docs are already amazing, as is the community here, and I am most grateful for both!