PageCreator fails to fetch nodes when cache is populated

[czabaj]

Description

We are using gatsby-plugin-page-creator in conjunction with gatsby-source-wordpress-experimental to create our FAQ's section.

On the WordPress side, we have a hidden page with WordPress Custom Fields which provides us with all FAQ data. We fetch this hidden page via gatsby-source-wordpress-experimental and in gatsby-node.js in onCreateNode, when the page with FAQ data is fetched, we create new GQL Nodes from it.

We obtain one JSON blob of all FaqCategory[] where FaqCategory contains also FaqQuestion[] and we create a flat FaqCategory and FaqQuestion GQL Nodes from it, linked together with createParentChildLink.

These new GQL nodes are used to create pages via new gatsby-plugin-page-creator FS routes API. We have a following folder structure

├── index.js
├── {FaqCategory.title}.js
└── {FaqQuestion.parent__(FaqCategory)__title}
    └── {FaqQuestion.question}.js

And it works great. Except, when we stop the dev-server and run it again, the develop always fails on sourcing the nodes with message:

PageCreator: Tried to create pages from the collection builder.
Unfortunately, the query came back empty. There may be an error in your query:

Cannot query field "allFaqCategory" on type "Query". Did you mean "allDirectory"?

We have some more pages generated from WordPress data and all of them fails, when we run dev-server with populated cache.

As a remedy, we run gatsby clean before each gatsby develop, so this is not a major issue, it just slows down starting of the dev-server, effectively prevents us from using the cache.

Steps to reproduce

In our (very complicated and, unfortunately, private) repo, run yarn develop, then kill the dev-server and run yarn develop again.

Expected result

Running yarn develop with populated cache starts normally.

Actual result

When the Gatsby cache is populated, the PageCreator fails to query GQL nodes created from WordPress data.

Environment

System:
OS: macOS 11.1
CPU: (8) x64 Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
Shell: 3.1.2 - /usr/local/bin/fish
Binaries:
Node: 12.19.0 - ~/.nvm/versions/node/v12.19.0/bin/node
Yarn: 1.22.10 - ~/.nvm/versions/node/v12.19.0/bin/yarn
npm: 6.14.8 - ~/.nvm/versions/node/v12.19.0/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Firefox: 85.0
Safari: 14.0.2

I have team of 4 and all of them have the same issue. Unfortunately, all of them have MacOS, so I cannot confirm weather it happens on another platform too.

Reproduction

I tried to reproduce this issue on minimal setup but it only occurs in our big repo. I can share at least the relevant code with you.

package.json

{
	"browserslist": [
		">0.25%",
		"Firefox ESR",
		"IE > 10"
	],
	"dependencies": {
		"@popperjs/core": "2.5.4",
		"@reach/router": "1.3.4",
		"@reduxjs/toolkit": "1.5.0",
		"apollo-cache-inmemory": "1.6.6",
		"apollo-client": "2.6.10",
		"apollo-link": "1.2.14",
		"apollo-link-context": "1.0.20",
		"apollo-link-error": "1.1.13",
		"apollo-link-schema": "1.2.5",
		"apollo-utilities": "1.3.3",
		"axios": "0.21.1",
		"classnames": "2.2.6",
		"core-js": "3.8.3",
		"final-form": "^4.20.1",
		"date-fns": "2.16.1",
		"fuse.js": "3.6.1",
		"gatsby": "2.31.1",
		"gatsby-image": "2.10.0",
		"gatsby-plugin-react-helmet": "3.9.0",
		"graphql-tag": "2.11.0",
		"is-iexplorer": "1.0.0",
		"jquery": "3.4.1",
		"js-cookie": "2.2.1",
		"jsonschema": "1.2.6",
		"moment": "2.29.1",
		"prettier-bytes": "1.0.4",
		"prop-types": "15.7.2",
		"react": "16.14.0",
		"react-animate-on-scroll": "2.1.5",
		"react-apollo": "3.1.5",
		"react-collapsed": "2.2.3",
		"react-copy-to-clipboard": "5.0.2",
		"react-dom": "16.14.0",
		"react-final-form": "~6.5.1",
		"react-helmet": "6.1.0",
		"react-image-file-resizer": "0.3.8",
		"react-intl": "3.12.1",
		"react-markdown": "4.3.1",
		"react-modal": "3.12.1",
		"react-onclickoutside": "6.9.0",
		"react-popper": "2.2.4",
		"react-redux": "7.2.2",
		"react-slick": "0.25.2",
		"react-spring": "^8.0.27",
		"react-tabs": "2.3.1",
		"react-transition-group": "2.9.0",
		"react-youtube": "^7.12.0",
		"reactour": "1.17.0",
		"recharts": "1.8.5",
		"recompose": "0.30.0",
		"redux": "4.0.5",
		"redux-form": "8.3.6",
		"redux-saga": "1.1.3",
		"reselect": "4.0.0",
		"scroll-into-view": "1.14.2",
		"slick-carousel": "1.8.1",
		"store2": "2.12.0",
		"styled-components": "4.0.0",
		"ua-parser-js": "0.7.23"
	},
	"devDependencies": {
		"@welldone-software/why-did-you-render": "^4.3.2",
		"apollo-link-logger": "1.2.3",
		"babel-plugin-transform-inline-environment-variables": "0.4.3",
		"babel-preset-gatsby": "0.11.0",
		"gatsby-cli": "2.18.0",
		"gatsby-plugin-advanced-sitemap": "1.5.7",
		"gatsby-plugin-extract-schema": "0.0.5",
		"gatsby-plugin-graphql-loader": "1.0.2",
		"gatsby-plugin-lodash": "3.9.0",
		"gatsby-plugin-manifest": "2.11.0",
		"gatsby-plugin-remove-trailing-slashes": "2.9.0",
		"gatsby-plugin-sass": "2.8.0",
		"gatsby-plugin-sharp": "2.13.1",
		"gatsby-plugin-svgr": "2.1.0",
		"gatsby-plugin-typescript": "2.11.0",
		"gatsby-source-filesystem": "2.10.0",
		"gatsby-source-wordpress-experimental": "5.0.2",
		"gatsby-transformer-inline-svg-v2": "1.1.0",
		"gatsby-transformer-sharp": "2.11.0",
		"mkdirp": "^1.0.4",
		"xmldom": "^0.3.0"
	},
	"private": true,
	"scripts": {
		"build": "cross-env NODE_ICU_DATA=../../node_modules/full-icu GATSBY_CONCURRENT_DOWNLOAD=10 gatsby build",
		"build:debug": "yarn build --no-uglify",
		"clean": "gatsby clean",
		"dev": "cross-env GATSBY_CONCURRENT_DOWNLOAD=5 WP_DISABLE_POLLING=1 gatsby develop -p 8080",
		"dev:profiling": "cross-env GATSBY_PROFILING=1 yarn dev",
		"dev:debug": "node --inspect-brk --no-lazy node_modules/.bin/gatsby develop -p 8080",
		"mock": "npx mockServer --outputDir src/spec --resolversDir mockResolvers --specFile ./openapi.yaml --staticFilesRoot ./__fixtures__/static",
		"mock:debug": "npx mockServer --outputDir ./src/spec --resolversDir mockResolvers --specFile ./openapi.yaml --debug",
		"serve": "gatsby serve"
	},
	"version": "4.16.0"
}

gatsby-config.js

const sass = require('sass');

require('dotenv').config({
	path: `.env.${process.env.NODE_ENV}`,
});

module.exports = {
	flags: {
		DEV_SSR: true,
		PRESERVE_FILE_DOWNLOAD_CACHE: true,
		PRESERVE_WEBPACK_CACHE: true,
	},
	plugins: [
		{
			resolve: `gatsby-source-wordpress-experimental`,
			options: {
				develop: {
					hardCacheData: true,
					hardCacheMediaFiles: true,
					nodeUpdateInterval: 30000,
				},
				production: {
					hardCacheMediaFiles: true,
				},
				type: {
					Category: { exclude: true },
					Comment: { exclude: true },
					ContentType: { exclude: true },
					MediaItem: {
						localFile: {
							maxFileSizeBytes: 10485760, // 10Mb
							requestConcurrency: 50,
						},
					},
					Menu: { exclude: true },
					MenuItem: { exclude: true },
					NodeWithContentEditor: { exclude: true },
					NodeWithExcerpt: { exclude: true },
					Post: {
						excludeFieldNames: [
							`categories`,
							`comments`,
							`content`,
							`enclosure`,
							`enqueuedScripts`,
							`enqueuedStylesheets`,
							`excerpt`,
							`postFormats`,
							`preview`,
							`revisions`,
							`revisionsOf`,
							`tags`,
						],
						limit: 10,
					},
					PostFormat: { exclude: true },
					Tag: { exclude: true },
					Taxonomy: { exclude: true },
					User: { exclude: true },
					UserRole: { exclude: true },
				}
			},
		},
		{
			resolve: `gatsby-plugin-extract-schema`,
			options: {
				dest: `${__dirname}/gatsby.schema.graphql.json`,
			},
		},
		`gatsby-plugin-graphql-loader`,
		{
			resolve: `gatsby-plugin-lodash`,
			options: {
				disabledFeatures: [`chaining`],
			},
		},
		{
			resolve: `gatsby-plugin-page-creator`,
			options: {
				path: `${__dirname}/src/pages`,
				ignore: [
					`**/_*/**/*`, // ignore subdirectories beginning with underscore
				],
			},
		},
		`gatsby-plugin-react-helmet`,
		`gatsby-plugin-remove-trailing-slashes`,
		{
			resolve: `gatsby-plugin-sass`,
			options: {
				implementation: sass,
				indentedSyntax: true,
			},
		},
		`gatsby-plugin-sharp`,
		{
			resolve: `gatsby-plugin-advanced-sitemap`,
			options: {
				exclude: [
					/\[[\w.]+\]\/$/, // filter out all dynamic pages created with FS API,
				],
			},
		},
		`gatsby-plugin-svgr`,
		`gatsby-plugin-typescript`,
		{
			resolve: `gatsby-source-filesystem`,
			options: {
				name: `assets`,
				path: `${__dirname}/src/assets`,
			},
		},
		`gatsby-transformer-inline-svg-v2`,
		`gatsby-transformer-sharp`,
	],
};

gatsby-node.js

const fs = require(`fs`).promises;

const _ = require(`lodash/fp`);
const mkdirp = require(`mkdirp`);
const path = require(`path`);

const wordpress = require(`./src/constants/wordpress.json`);
const webpackConfig = require(`./webpack.config`);

const removeWebpackLoaderForPdfFiles = _.update(
	`module.rules`,
	_.remove((rule) => rule.test && _.isFunction(rule.test.test) && rule.test.test(`.pdf`))
);

/**
 * Enhance Gatsby webpack config with custom configs
 * @see https://www.gatsbyjs.org/docs/node-apis/#onCreateWebpackConfig
 * @see https://www.gatsbyjs.org/docs/actions/#setWebpackConfig
 */
exports.onCreateWebpackConfig = function onCreateWebpackConfig({
	actions,
	getConfig,
	loaders,
	stage,
}) {
	const config = getConfig();
	const newConfig = removeWebpackLoaderForPdfFiles(config); // is replaced in webpack.config.js
	actions.replaceWebpackConfig(newConfig);
	actions.setWebpackConfig(webpackConfig);

	// Remove SSR incompatible modules during prerender
	if (stage === 'build-html') {
		actions.setWebpackConfig({
			module: {
				rules: [
					{
						test: /reactour/,
						use: loaders.null(),
					},
				],
			},
		});
		actions.replaceWebpackConfig(_.omit([`externals`], getConfig()));
	}
	if (!process.env.GATSBY_PROFILING) {
		actions.setWebpackConfig({
			module: {
				rules: [
					{
						test: /@welldone-software\/why-did-you-render/,
						use: loaders.null(),
					},
				],
			},
		});
	}
};

// silently export "createWellKnown" for the test, creating separate "__testExports" wont work here, as Gatsby checks
// all exported members of this file and warns on unknown exports
exports.onPostBuild.createWellKnown = createWellKnown;

// WordPress plugin Kiwi appends HTML code of social share buttons to wysiwyg text fields. This is not intended on our
// web, so we strip this HTML from the WordPress content (the plugin can be disabled per-page only in paid version).
const wordPressKiwiPluginHtmlRe = /<ul class="kiwi-article-bar.*<\/ul>/;
/**
 * Creates GraphQL nodes from Wordpress CMS data. This is only needed for CMS data which generates pages via
 * gatsby-plugin-page-creator. Other CMS data could be simply loaded via `useStaticQuery` hook.
 */
// eslint-disable-next-line consistent-return
exports.onCreateNode = function onCreateNode({
	createContentDigest,
	createNodeId,
	node,
	actions: { createNode, createParentChildLink },
}) {
	// eslint-disable-next-line default-case
	switch (node.internal.type) {
		case `WpPage`:
			// eslint-disable-next-line default-case
			switch (node.databaseId) {
				case wordpress.faqPageId:
					return Promise.all(
						node.faq.category.map(async (faqCategory) => {
							const faqCategoryNode = {
								icon___NODE: faqCategory.icon.id,
								id: createNodeId(faqCategory.title),
								internal: {
									type: `FaqCategory`,
									mediaType: `application/json`,
									contentDigest: createContentDigest(faqCategory),
								},
								parent: null,
								title: faqCategory.title,
							};
							await createNode(faqCategoryNode);
							return Promise.all(
								faqCategory.questions.map(async (faqQuestion) => {
									const faqQuestionNode = {
										answer: faqQuestion.answer,
										id: createNodeId(faqQuestion.question),
										internal: {
											type: `FaqQuestion`,
											mediaType: `application/json`,
											contentDigest: createContentDigest(faqQuestion),
										},
										keywords: faqQuestion.keywords,
										parent: faqCategoryNode.id,
										question: faqQuestion.question,
									};
									await createNode(faqQuestionNode);
									return createParentChildLink({ parent: faqCategoryNode, child: faqQuestionNode });
								})
							);
						})
					);
				case wordpress.pressReleasesPageId:
					return Promise.all(
						node.pressRelease.pressRelease.map((pressRelease) =>
							createNode({
								date: pressRelease.date,
								file___NODE: pressRelease.file.id,
								id: createNodeId(pressRelease.title),
								internal: {
									type: `PressRelease`,
									mediaType: `application/json`,
									contentDigest: createContentDigest(pressRelease),
								},
								parent: null,
								perex: pressRelease.perex,
								text: pressRelease.text.replace(wordPressKiwiPluginHtmlRe, ``),
								title: pressRelease.title,
							})
						)
					);
			}
			break;
	}
};

[LekoArts]

Hi!

We'll need a minimal reproduction for this. However, I have an idea what it might be:

https://www.gatsbyjs.com/docs/node-creation/#freshstale-nodes

Any nodes that aren’t touched by the end of the source-nodes phase, are deleted. This is performed via a diff between the nodesTouched and nodes Redux namespaces, in source-nodes.ts

On your first run without a cache all nodes are "fresh" so they are all created. But on further runs your newly created nodes inside onCreateNode are probably not "touched" and thus garbage collected.
This can happen when you e.g. don't set the parent when it's necessary.

The source plugin tutorial also touches (heh) on this API:
https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-source-plugin/#create-remote-file-node-from-a-url

[czabaj]

@LekoArts fantastic. I have put the code

// gatsby-node.js

exports.sourceNodes = ({ actions, getNodesByType }) => {
	const { touchNode } = actions;
	// touch nodes to ensure they aren't garbage collected
	[
		...getNodesByType(`FaqCategory`),
		...getNodesByType(`FaqQuestion`),
	].forEach((node) => touchNode({ nodeId: node.id }));
};

to the test and so far, it seems to resolve the issue.

Thank you for the link to the "Creating a Source Plugin", this is the part of the documentation, which I not yet discovered. I will read it through and through.