Incremental Build

[lschierer]

Coming out of #1485, #682, #1402, #1493 and some of the other many bugs I've either submitted or commented on recently, it occurred to me that Greenwood would benefit from an incremental build mode similar to that tsc optionally has.

psuedo logic:

1. if the output file in public is newer than ( the layout && the file in src/pages )
2. scan the file for imports
3. if the output file is newer than the output files that correspond to each import
4. do not rerun the render cycle, it will result in the same page.
5. if any of the above tests fail, run the render cycle
6. optionally keep track of this in graph.json similar to the way tsc generates the tsbuildinfo files

My thought is that I have a project with 886 entries in graph.json 25 of which have isSSR set, meaning the vast majority of them should render based on the markdown + the layout (and any components it includes) + any plugins that intercept it. Even though its mostly static, on a reasonably performant desktop it takes an hour to run the build.

there are a few things that seem to behave differently between dev mode and when actually deployed. Some of those are because I can deploy the site any number of ways (Greenwood has hosting examples for 5 different ways each of which will behave slightly differently) and dev mode can't be expected to reasonably replicate the nuances of all of them. So there are some things that I discover aren't quite what I expect when I deploy the first time. Or maybe its my infrastructure as code itself that I'm trying to debug. In either case, I do not want an hour build between iterations.

Sometimes of course that can't be helped. If I change a core CSS file or a heavily used layout file, then even with an incremental mode, nearly everything will recompile. But if I am trying to get an API endpoint to do exactly what I need in my hosting environment, or if I am testing how SST (which wants to run the build command itself) deploys the site (that example I mentioned above where its my infrastructure as code that's being tested, not the site itself) then an incremental mode could save vast amounts of time.

[thescientist13]

Yeah, long build times are definitely no fun while trying to iterate, though it's definitely surprising to hear of hour long build times like you experienced. I would say for sure evaluating some of the reproductions you provided in #1485 will definitely be a good first start for us, which I have made sure to track as part of #970 , which I plan to conduct another run through and probably flesh out more as well heading towards an eventual 1.0 release and getting a bit more familiar with Node / Chrome profiling side of things. In those older Greenwood versions with some light prerendering and thousands of markdown pages, we at least managed to get over the finish line in a few minutes (on an M1 air, so circa 2020), so will take your reproductions and layer those on top to see what's going on! 🔍

So once we've made sure there's as high as throughput possible for the first build pass, then I'm sure evaluating something like how Turborepo and Nx do things like incremental builds would be valuable in pursing a similar technique for Greenwood, which would likely provide an easy on ramp to take this feature from build time to runtime, a la Vercel's ISR (Incremental Static Regeneration)

[lschierer]

Its worth noting that my build times are most definitely not on a bunch of light markdown pages. They aren't SSR in the sense they are all pre-renderable, but I'm definitely taking advantage of Greenwood's capabilities. Nearly every page is hitting a footer plugin, a header plugin, using a sidebar component, and many of them are loading at least one other component. There is probably still quite a bit I could do to optimize the behavior of both plugins, and I already moved at least some of the components out of greenwood into scripts that are run from the assets directory prior to build orchestrated by the justfile. There's definitely a per-plugin performance hit, though I don't have good scientific numbers on it, and so removing the external source plugins (plural - I at one point had one for each kind of file being converted) and replacing them with scripts that justfile runs helped a fair amount. That helped me because my "external" source is a bunch of local yaml and json files, but obviously wouldn't help people using a more dynamic CMS as their external source.

I also ended up giving SST a fake build command - essentially just an echo statement, trusting just to have already built the site before SST is called, because SST wasn't handling the long build time well independently of the cost to iterate while testing my infrastructure as code itself being too high.

So while I do very much still want an incremental build, I think I've acceptably worked around not having one.

[lschierer]

Not to sound utterly crazy or as if I were contradicting myself, but I added a page tonight, and the build took

  3673.52 real     14794.75 user      1555.84 sys
         13869350912  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
             4052624  page reclaims
                 526  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
               10638  messages sent
               10803  messages received
                  94  signals received
                7567  voluntary context switches
           163281890  involuntary context switches
           147101692  instructions retired
           128781667  cycles elapsed
             3130304  peak memory footprint

per /usr/bin/time -l (I forgot to use the -h flag)