It spent 10 minutes and 2GB RAM to update 5 pages form an total of 100 000 pages

[nuxtdev]

Description

It spent 5-10 minutes and 2GB RAM to update 5 pages form an total of 100 000 pages!
I think it will be more smart and not check all pages!

GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages

success open and validate gatsby-configs - 0.159s
success load plugins - 1.932s
success onPreInit - 0.122s
success initialize cache - 0.017s
success copy gatsby files - 0.190s
success onPreBootstrap - 0.045s
success createSchemaCustomization - 0.013s
success Checking for changed pages - 0.003s
success source and transform nodes - 47.020s
success building schema - 1.824s
info Total nodes: 113470, SitePage nodes: 1 (use --verbose for breakdown)
success createPages - 0.011s
success Checking for changed pages - 0.002s
success createPagesStatefully - 308.983s
success Cleaning up stale page-data - 9.345s
success update schema - 1.667s
success onPreExtractQueries - 0.014s
success extract queries from components - 4.250s
success write out redirect data - 0.007s
success Build manifest and related icons - 0.251s
success onPostBootstrap - 0.280s
info bootstrap finished - 391.081s
success run page queries - 2.266s - 5/5 2.21/s
success write out requires - 0.474s
success Building production JavaScript and CSS bundles - 18.685s
success Building HTML renderer - 6.731s
[                            ]   0.001 s 0/5 0% Building static HTML for pages
success Building static HTML for pages - 0.417s - 5/5 11.99/s
success onPostBuild - 7.989s
info Done building in 449.669603158 sec
info Built pages:
Updated page: /
Updated page: /mysql-products-7/
Updated page: /mysql-products-8/
Updated page: /mysql-products-9/
Updated page: /mysql-products-10/

Steps to reproduce

Clear steps describing how to reproduce the issue. Please link to a reproduction, this makes your issue much easier to diagnose (seriously).

How to Make a Minimal Reproduction: https://www.gatsbyjs.com/contributing/how-to-make-a-reproducible-test-case/

Expected result

What should happen?

Actual result

What happened.

Environment

Run gatsby info --clipboard in your project directory and paste the output here.

[nuxtdev]

And if no page to update :)

GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages

success open and validate gatsby-configs - 0.173s
success load plugins - 2.099s
success onPreInit - 0.217s
success initialize cache - 0.026s
success copy gatsby files - 0.610s
success onPreBootstrap - 0.064s
success createSchemaCustomization - 0.028s
success Checking for changed pages - 0.004s
success source and transform nodes - 56.590s
success building schema - 1.879s
info Total nodes: 113470, SitePage nodes: 1 (use --verbose for breakdown)
success createPages - 0.007s
success Checking for changed pages - 0.002s
⠙ createPagesStatefully

success createPagesStatefully - 320.243s
success Cleaning up stale page-data - 10.456s
success update schema - 1.399s
success onPreExtractQueries - 0.009s
success extract queries from components - 4.211s
success write out redirect data - 0.009s
success Build manifest and related icons - 0.267s
success onPostBootstrap - 0.298s
info bootstrap finished - 414.132s
success write out requires - 0.475s
success Building production JavaScript and CSS bundles - 21.783s
success Building HTML renderer - 5.667s
info There are no new or changed html files to build.
success onPostBuild - 9.425s
info Done building in 474.585669695 sec

[nuxtdev]

Read data from mysql like is say in your doc

[LekoArts]

What's your question here? Do you keep the .cache and public folder around when rebuilding the site?

[nuxtdev]

Yes I think I keep.
I think the problem is in DB, in 100 000 rows.
I think gatsby read all 100 000 rows to recreate only one page or 0!

---

How it work? Read each time all data from DB to grapql?
With what it spend that time, 320.243s?

[KyleAMathews]

It sounds like the source plugin doesn't know how to only fetch data that's changed since the last run. Gatsby doesn't fetch data itself — it's the source plugin you're using that does this and needs to be smarter.

[nuxtdev]

gatsby eat 2 GB of memory and 10 min of my life to make that check and determine what no page to recreate :) :).

[nuxtdev]

Very nice.

Conclusion - It's the end!!!!!

gatsby - hard recreate modified pages if need to create one page form 100 000, also it spend very much time to check that it has nothing to update :) lol ??

next - its incremental shit si shit, or I not understand or it make a cache of page for some seconds and serve it until it is not expired, after it recreate page and serve it to next user. If 1000 will try to acces 1000 different pages in the ame time it will render 1000 pages in same time = out of memory :) lol ??

nuxt - it not have such shit, but can be added swr cache module what will have the same efect like next incremental shit :) lol ??

It's the end.

I'm gonna use SPA mode and fuck SEO and other shit

[LekoArts]

Sorry, I still don’t understand what you are trying to tell us. If you expect less pages to be recreates you might use some patterns that disallow this: https://www.gatsbyjs.com/docs/debugging-incremental-builds/

[nuxtdev]

I make a build with 100 000 pages and have 100 000 folders in public dir.
I delete from mysql all 100 000 rows and add 2 000 new rows (with other id start from 101 000 for example)
Run GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
See in public dir 120 000 folders not 2 000
I will try it again later.

[nuxtdev]

Sorry, I still don’t understand what you are trying to tell us. If you expect less pages to be recreates you might use some patterns that disallow this: https://www.gatsbyjs.com/docs/debugging-incremental-builds/

how you make gatsbyjs if you understand nothing