Building large site (~50K) using template and JSON Data

[sonalimerchia]

I have a website of about 50K pages that are mostly just templates from an array in a central json file. It's currently taking a really long time to build so netlify is timing out.

The data.json file is just an array of objects while gatsby-node looks kind of like this:

exports.createPages = async ({ actions }) => {
   const { createPage } = actions;

   const data = require('data.json');
   for (const obj  of data) {
      createPage({
         path: obj.path, 
         component: path.resolve(__dirname, 'src/templates/dataTemplate.tsx'),
         context: obj
      })
   }
}

The template doesn't have any queries and the step in the build that takes too long is the Building static HTML for pages

I'm not sure how to go about improving the build time. Is there a better way to do this?

[KyleAMathews]

Does it fail locally too?

[sonalimerchia]

No but it takes like 15 minutes. On Netlify it takes over 30 so the build fails.

[KyleAMathews]

Ah, I think Netlify just times out at 15 minutes. So this isn't Gatsby related. Gatsby Cloud doesn't have a timeout like this so you could try building there.

[sonalimerchia]

Our Netlify build is upgraded so it fails after 30. Plus, apparently there are people with much larger sites that build much faster so I was wondering if I was missing anything

[KyleAMathews]

How fast a page takes to render varies a great deal depending on your component tree. Deeper trees & complicated components are slow. "Hello World" pages, for example, will build at 3-5k pages / second so 50k pages would take only 10s. You can use the React profiler to see if there's any quick wins to be had.