Gatsby Cloud preview is very slow (3 min plus) using gatsby-source-contentful plugin

[deconstructionalism]

Description

We just started using Gatsby Cloud on a trial run with our organization. We host 600+ static sites with all of the content being modeled and persisted within Contentful. We recently began re-tooling our querying for site data to try and take advantage of Cloud Previews.

The way we use Contentful is that we have a top-level content model called Website where each site has a unique key called siteId. All of the site data is joined into the Website content instance via content relations. For a given instance of the app, we only need the data for one site via it's siteId. IE, each Gatsby app instance serves previews for one site.

When our Gatsby app builds, the gatsby-contentful-source plugin gets the data for the entire space on Contentful, ie 600+ sites and all of their content. This takes a long time for sourcing, fetching, syncing and building. For builds, this isn't so much an issue as we do not need published edits to show near-real-time changes in the published site.

However, we do want near-real-time updates to the preview sites and it seems that the Preview builds spend a lot of time processing data, which is the bottleneck step for time efficiency. You can see that the Preview re-build to ~1 min on Gatsby Cloud for a change in one text field:

success Contentful: Process data (rt5zmd3ipxai-arjun-template-prototype) - 62.935s
info Creating 1051 Contentful Hero Card nodes
info Creating 744 Contentful Hero carousel nodes
info Creating 3931 Contentful Link nodes
info Creating 641 Contentful Website nodes
info Creating 591 Contentful Review Blade nodes
info Creating 1542 Contentful Service Details Card nodes
info Creating 282 Contentful Override Background nodes
info Creating 8 Contentful 60-40 Panel Blade nodes
info Creating 1 Contentful Template - Services Directory 3.0 nodes
info Creating 383 Contentful Documents and Forms nodes
info Creating 366 Contentful Online Appointment iFrame nodes
info Creating 63 Contentful Blog Category nodes
info Creating 13 Contentful Location Blade - Multiple nodes
info Creating 4978 Contentful Services Blade - Block Listing nodes
info Creating 11 Contentful Location Information nodes
info Creating 508 Contentful Gallery nodes
info Creating 475 Contentful 3-Column Text Blocks nodes
info Creating 13 Contentful Jump Link nodes
info Creating 6 Contentful Jump Link Group nodes
info Creating 81 Contentful Accordion Blade nodes
info Creating 791 Contentful Accordion nodes
info Creating 10342 Contentful Staff nodes
info Creating 2271 Contentful Staff Blade nodes
info Creating 6 Contentful Staff Group nodes
info Creating 1 Contentful Staff Accreditation nodes
info Creating 68 Contentful Image Slider nodes
info Creating 1 Contentful Services Blade - Detailed Listing nodes
info Creating 1472 Contentful Service Details nodes
info Creating 890 Contentful Rich Text Blob nodes
info Creating 93 Contentful Blog Article nodes
info Creating 23 Contentful Pencil Banner nodes
info Creating 1210 Contentful Secondary Hero nodes
info Creating 163 Contentful Key Category Menu nodes
info Creating 19 Contentful Item Label nodes
info Creating 171 Contentful Key Category Menu Group nodes
info Creating 616 Contentful Website Header nodes
info Creating 654 Contentful Location Information Blade nodes
info Creating 1910 Contentful Promo Banner nodes
info Creating 1765 Contentful Review Card nodes
info Creating 609 Contentful Website Footer nodes
info Creating 784 Contentful Social Links Container nodes
info Creating 2169 Contentful 60-40 Panel nodes
info Creating 585 Contentful 5-Card Category Cards Blade nodes
info Creating 1067 Contentful 5-Card Category Card nodes
info Creating 488 Contentful 3-Tile Promotion Cards Blade nodes
info Creating 892 Contentful 3-Tile nodes
info Creating 134 Contentful Video nodes
info Creating 19685 Contentful Image nodes
info Creating 475 Contentful Custom Code nodes
info Creating 5539 Contentful Website Page nodes
info Creating 24 Contentful Story Blade nodes
info Creating 23 Contentful Blog Blade nodes
info Creating 620 Contentful Location Container nodes
info Creating 597 Contentful FormKeep Form nodes
info Creating 25339 Contentful asset nodes

• Is there a way to get previews to be faster without completing re-factoring our content model?
• Or otherwise, how would you advise Contentful-sourced apps with hundreds of sites to take advantage of Preview builds?

Steps to reproduce

This is more a question content architecture in relation to Gatsby Cloud architecture. We can't really advise on how to repro this unless you have access to our deeply nested content model.

Expected result

Near-instant preview build updates.

Actual result

Preview build updates took over a minute.

Environment

  System:
    OS: macOS 11.2.1
    CPU: (4) x64 Intel(R) Core(TM) i7-7660U CPU @ 2.50GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.16.1 - ~/.nvm/versions/node/v14.16.1/bin/node
    Yarn: 1.5.1 - /usr/local/bin/yarn
    npm: 6.14.12 - ~/.nvm/versions/node/v14.16.1/bin/npm
  Languages:
    Python: 3.7.9 - /Users/arjunray/.pyenv/shims/python
  Browsers:
    Edge: 90.0.818.51
    Firefox: 62.0
    Safari: 14.0.3
  npmPackages:
    gatsby: ^3.4.0 => 3.4.0
    gatsby-plugin-image: ^1.4.0 => 1.4.0
    gatsby-plugin-sharp: ^3.4.0 => 3.4.0
    gatsby-source-contentful: 5.5.0-next.0 => 5.5.0-next.0
    gatsby-transformer-remark: ^4.0.0 => 4.1.0
    gatsby-transformer-sharp: ^3.0.0 => 3.4.0

[deconstructionalism]

Also, I've been looking at the config for the gatsby-source-contentful plugin and I don't see any way to segment the data such that only one siteId's data is fetched. Am I missing something or is this not actually possible to do with the plugin?

[KyleAMathews]

Huh wow. For fun, how many nodes are being created?

Tagging @axe312ger.

He would know if contentful's API allows for filtering your content to just the content for a single site. If so, seem pretty reasonable to support that in the source plugin.

[deconstructionalism]

@KyleAMathews

So this line here could be modified in order to sync a specific content instead of all content:

gatsby/packages/gatsby-source-contentful/src/fetch.js

Line 230 in 4ecf00a

const basicSyncConfig = {

by using the content_type option as documented here.

I'm going to try some performance measurements using the data in my space with and without setting the content_type to site (the Website content type in our space).

[deconstructionalism]

The old version of our app did query using the generic gatsby-source-graphql plugin. It looks like they were not restricting the query by siteId so perhaps we could reduce query times that way

[deconstructionalism]

@axe312ger @KyleAMathews

Also, is it correct that incremental builds and preview builds for Contentful sourced apps are only available on Gatsby Cloud if we use the gatsby-source-contentful plugin?

[deconstructionalism]

do you need to use gatsby-source-contentful to have access to Gatsby Cloud previews and incremental builds?

[KyleAMathews]

No it works with any source plugin. You could e.g. try publishing your version of gatsby-source-contentful with the change to NPM and deploy it to Gatsby Cloud to see how it works.

[axe312ger]

@deconstructionalism

Using a single space for 600 websites is far from the intended use. Contentful wants you to use one space per project. That is how the (sync) API works as well. We can't filter entries, we just get all of them.

That means even when we would want to support that uncommon use, we can not.

Please split up your space into a space per website/project:

You can use https://www.contentful.com/developers/docs/tutorials/cli/scripting-migrations/ and https://www.contentful.com/faq/content-migrations/ to keep the content model in sync.

You can also reach out to https://www.contentful.com/support/. They might be able to help you with splitting up the space.