Build Performance

[ScyDev]

Gatsby noob here, needing some feedback on what I can expect, what's normal of very poor performance.

We're working on a shop that should have 2'000'000 products when finished. Currently developing with 50'000 products (one image each) and a Gatsby (v3.3) build is taking 2-3 hours. (I have tried running it on a super fast server with 48 cores, made only a small difference)

I'm using Strapi as the backend. The phase of getting data from Strapi is quite fast. But building the pages is what takes so long.

The frontend is absolutely minimal: building all individual product pages and building a paginated list for all 50k products. For 2 languages.

Is this expected behaviour? Or what are your experiences with similar amounts of data and build time?

Also, I'm running into ERROR
Failed to write out requires Maximum call stack size exceeded
RangeError: Maximum call stack size exceeded
when going above 50k products.
Does someone know what the cause of this is? (the only answers I found so far are "use newer Gatsby version", but no indication as to what causes it)

Thanks for any insights!

[westhouseit]

Hi

Our current site has about 5k pages sourced from a Directus API and takes about 13mins to build on Netlify.
We've run into issues with building the remaining about 500k pages and so are currently using client-side routes for those pages. I wonder whether it's possible that the GraphQL schema is too large and so is running out of memory. After completing some other work I'd like to go back and look at possibly skipping the GraphQL import - https://www.gatsbyjs.com/docs/how-to/querying-data/using-gatsby-without-graphql/

Any progress on your end?

P.S. Also check these if you haven't already: https://www.gatsbyjs.com/docs/scaling-issues/

Edit: And also these suggestions might help: https://www.netlify.com/blog/2020/06/11/5-optimizations-for-faster-gatsby-builds/

[KyleAMathews]

Maximum call stack size exceeded happens when there's some code in core or a plugin that hasn't been tested at this scale. E.g. simple loops over all your pages don't work any more like #31547

Does the error say where in the code the "maximum call stack" problem is hit?