gatsby build is killed on CI because it takes too long and uses too much memory

[jonhue]

Description

When running gatsby build both locally and on GitHub Actions, the process takes a lot of memory (locally ~14GB) and is killed eventually. The development builds are building fast and work fine.

Steps to reproduce

This error can be reproduced with https://github.com/bifolia/frontend. An example of this behavior can be seen in the following logs: https://github.com/bifolia/frontend/runs/2272841602

Expected result

The build should not be killed.

Actual result

The build is killed.

Environment

  System:
    OS: Linux 4.19 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 15.13.0 - /tmp/yarn--1617656412419-0.45431593089297895/node
    Yarn: 1.22.5 - /tmp/yarn--1617656412419-0.45431593089297895/yarn
    npm: 7.7.6 - ~/.asdf/installs/nodejs/15.13.0/bin/npm
  Languages:
    Python: 2.7.18 - /usr/bin/python
  npmPackages:
    gatsby: ^3.2.1 => 3.2.1
    gatsby-plugin-gatsby-cloud: ^2.2.0 => 2.2.0
    gatsby-plugin-google-gtag: ^3.2.0 => 3.2.0
    gatsby-plugin-image: ^1.2.0 => 1.2.0
    gatsby-plugin-manifest: ^3.2.0 => 3.2.0
    gatsby-plugin-offline: ^4.2.0 => 4.2.0
    gatsby-plugin-react-helmet: ^4.2.0 => 4.2.0
    gatsby-plugin-sass: ^4.2.0 => 4.2.0
    gatsby-plugin-sharp: ^3.2.0 => 3.2.0
    gatsby-source-filesystem: ^3.2.0 => 3.2.0
    gatsby-source-strapi: ^1.0.0-alpha.0 => 1.0.0-alpha.0
    gatsby-transformer-sharp: ^3.2.0 => 3.2.0

[LekoArts]

You're using a bad pattern in your site that is the cause of the issue: https://www.gatsbyjs.com/docs/creating-and-modifying-pages/#performance-implications

Instead of querying everything in gatsby-node.js and passing node in context:

https://github.com/bifolia/frontend/blob/dd538cbc191e517f621b2924cda51f63f6bacafd/gatsby-node.js#L58-L109

You should only pass a unique identifier and use that as a GraphQL variable in the query of the template.

[LekoArts]

You are querying for huge images everywhere (I only picked this example but saw it in nearly every query: https://github.com/bifolia/frontend/blob/10f73eb23246f72b163a702136eadc2e1e534a35/src/templates/projekt.tsx#L55). Sharp can be memory hungry for such huge images. Do you actually want to serve such huge images?

[jonhue]

I reduced the image sizes to a maybe more reasonable 1920px, but am still running into the error: https://github.com/bifolia/frontend/runs/2276987423.

One other thing that makes me wonder is that this happens at the very end of the build process when building the static html pages is the last remaining step. Do we perform some specific expensive operations at that stage?

[LekoArts]

Also try removing AVIF from the image query. The GitHub action runner might be too weak for that, too. sharp's AVIF integration currently is more power hungry than the other formats, too

[jonhue]

Awesome seems like that was the culprit! Thank you for your help!

[LekoArts]

Cool! If you want to use AVIF you'll then need to look out for another build service (https://www.gatsbyjs.com/cloud/ allocates enough memory for example) or wait until it gets more efficient in sharp (it's still early days)