Restrict build to new and/or updated pages?

[Wysardry]

Is there currently a way to only build pages that are new or updated?

If not, could such a feature be added?

I would like to use a local installation of Wagtail CMS to build static HTML pages to upload to a cheap shared hosting package via FTP. Having to upload all the HTML files would gradually become more of a pain as the site grew larger.

[loicteixeira]

Sadly it isn't as straightforward as only building new or updated pages. A few example come to mind:

• Given a blog index page, and some article pages as children. The index page probably need rebuilding (so it display an updated list of links to the child pages) but that page actually wasn't updated per-se.
• Given a header with navigation links showing the title of a page, renaming one of the page title should update the menu, and therefore all the pages, despite those pages not being "updated"
• What about deleted pages?

As a workaround for now, I think you should be able to build the site as normal, then use rsync --checksum --dry-run other-parameters-here between the build folder and an intermediary folder to get a list of modified files, and then somehow pick them up and only upload those (or maybe your FTP utility can do that?).

Really rough experiment:

# Setup
$ mkdir {src,dest}

$ echo "hello" > src/{foo,bar,baz}.txt

$ ls -al dest src
dest:
total 0
drwxr-xr-x  2 loic  staff   64 Aug  3 17:53 .
drwxr-xr-x  4 loic  staff  128 Aug  3 17:53 ..

src:
total 24
drwxr-xr-x  5 loic  staff  160 Aug  3 17:54 .
drwxr-xr-x  4 loic  staff  128 Aug  3 17:53 ..
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 bar.txt
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 baz.txt
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 foo.txt

# First sync
$ rsync --archive --delete src/ dest

$ ls -al dest src
dest:
total 24
drwxr-xr-x  5 loic  staff  160 Aug  3 17:54 .
drwxr-xr-x  4 loic  staff  128 Aug  3 17:53 ..
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 bar.txt
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 baz.txt
-rw-r--r--  1 loic  staff    6 Aug  3 17:54 foo.txt

src:
total 16
drwxr-xr-x  4 loic  staff  128 Aug  3 17:55 .
drwxr-xr-x  4 loic  staff  128 Aug  3 17:53 ..
-rw-r--r--  1 loic  staff    6 Aug  3 17:55 bar.txt
-rw-r--r--  1 loic  staff   13 Aug  3 17:55 foo.txt

# Making some changes
# - Append more test to `foo.txt`
# - Update the "modified time" of `bar.txt` (but the content stay the same)
# - Delete `baz.txt`
$ echo " world" >> src/foo.txt; touch src/bar.txt; rm src/baz.txt

# Now use rsync in dry-run mode to get some information of what rsync would do
$ rsync --archive --delete --checksum --dry-run --itemize-changes src/ dest
*deleting baz.txt
.d..t.... ./
.f..t.... bar.txt
>fcst.... foo.txt

We use:

• --checksum so it only check the size of the items (if it's different, it's considered for sync, if it's the same, it does a checksum to be extra sure), this allows to rule out files which have been rebuilt with the exact same content.
• --dry-run so it doesn't actually do anything
• --itemize-changes so it prints the files and some information.

So if the line starts with:

• *deleting, well it would have been deleted
• >, then it would have been updated (the rest of the letters and dots indicates why, but we don't care here)

So we can try to save this information to disc, and then run some more scripting magic to clean up the list:

$ rsync --archive --delete --checksum --dry-run --itemize-changes src/ dest > rsync.log

$ grep '*deleting' rsync.log | cut -d" " -f2 > file-to-delete.log

$ grep '^>' rsync.log | cut -d" " -f2 > files-to-update.log

$ cat files-to-update.log 
foo.txt

[Wysardry]

There are existing static site generators that only build pages that have been changed, so there must be ways around the issues you mentioned. Nikola (also in Python) is one and Eleventy (NodeJS) is another.

There's usually an option to rebuild the entire site if you know you've made a global change. Auto generated blog indexes, site maps etc. are usually rebuilt every time, regardless of whether you ask for a full or partial rebuild.

At this point, I'm still exploring potential solutions for creating a number of websites. I'm reasonably sure I will be using Nikola for a small website (under 50 pages) and was considering using Wagtail for a larger website (several hundred pages).

Since my initial post, I did find a utility called FreeFileSync that might help, although I haven't tested it. It should reduce the number of files that are uploaded, but wouldn't lower the number of files that are actually built.

One of the reasons I would like to build fewer files each time is to reduce wear and tear on my storage drives. The only way I can think of to reduce the chances of failure is to build the pages on a USB thumb drive and replace it on a regular basis.

[loicteixeira]

There are existing static site generators that only build pages that have been changed, so there must be ways around the issues you mentioned.

Yes, it's certainly doable, but not necessarily straightforward.

There's usually an option to rebuild the entire site if you know you've made a global change.

I'd be wary about such option, because it means that you have to know whether your change was global or not, which might not be something that Wagtail editors (users of the final site) might be able to grasp too easily. Now I think wagtail-bakery is a bit of a niche project, so the users implementing it might also be the users using it, but still.

Auto generated blog indexes, site maps etc. are usually rebuilt every time, regardless of whether you ask for a full or partial rebuild.

Wagtail doesn't really have a concept of an index page. And trying to define it as for example "a page which has children" might not capture everything. So that would leave it as some sort of flag to set on a page model?

---

Don't get me wrong, I'd love to see this feature in Wagtail-Bakery, but given I'm not able to spend too much time on the project and that there's already many things we should fix/implement to get to a proper 1.0 version, it might be quite some time before incremental builds is implemented.

[laymonage]

I think it'd be extremely hard to implement this feature, at least without asking website implementers to write hints to let wagtail-bakery know the dependencies that a page has. The reason for this is, as @loicteixeira mentioned, is that relationships between pages are arbitrary and not always defined through database semantics (e.g. ForeignKey).

In a given page, you may have parts of content that comes from other pages that may or may not be related via a ForeignKey, because you have completely dynamic control (defined through code) over the page rendering mechanism. The navigation header example written above is a good example of this. Another example to highlight the "dynamic" nature is a page that relies on third-party APIs (interacted with in Python, on the server), where the content may change at build time.

As far as I know (correct me if I'm wrong), the tools you mentioned (Nikola and Eleventy) can do this because they are inherently static site generators. This means that the pages are all stored in the source code, and any relations are explicitly defined in the source code of each page through something like e.g. {% include "other_page.html" %} or {% extends "some_other.html" %} that can all be analysed at build time without executing user-written code (templates and configuration files are not executable). By analysing the project code and comparing it to the previous build cache, it can make the necessary "incremental" optimisations.

Meanwhile, Wagtail is inherently dynamic, and wagtail-bakery "bakes" a snapshot of the website into a static website. It cannot know the dependencies of pages without running the code that renders them, at which point you're basically rebuilding the whole website.

For this to be feasible, you'll need to have an API for defining the dependencies of a page so that wagtail-bakery knows which pages to update, and it is up to the user to define the dependencies (defining incorrect dependencies could result in stale pages).

An example of this is Next.js's revalidate mechanism for its Incremental Static Regeneration feature: https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration

That said, even in Next.js the routing of your website is defined through your source code structure, so it has the advantage of knowing the URL patterns just by looking at the source code. In Wagtail, with the same source code, the routing can be completely different depending on the state of the database.

So yeah, it's possible, but not without the user explicitly defining the dependencies.

[Wysardry]

I'm not sure how Eleventy decides what pages to rebuild, but with Nikola and Pelican there are two different document types: web page and blog post. These are built from markdown files stored in appropriately named directories.

It is up to the user to create index files in the "pages" directory, but the script will automatically build them for documents in the "posts" directory.

You can also (optionally) set the page type to "page" or "post" in the metadata part of each markdown file which will allow you to store your markdown files in any directory you wish. Where the final pages are built can also be set using metadata or a configuration file.

I would expect Wagtail Bakery to also need to be told which pages should always be rebuilt and which should only be built when changed. I don't really have a preference for how or where this is defined. It could be explicitly set in the metadata for each document or based on what document type or template is used.

Command line static site generators don't usually have user roles, so I didn't really consider how they would fit in with the baking process. However, I would imagine that anyone (on a multi-user system) who had permissions allowing them to bake the site would have at least a basic idea of which changes would have a global impact.

So, is this a feature that might be added eventually or definitely not?

[laymonage]

It's still within the realms of possibility, and would be nice to have in wagtail-bakery, but I wouldn't expect it to be done any time soon. Unless someone's willing to do it and send a PR, of course 😄