Gatsby builds fail when an image (that's not being used) is missing.

[mcmadeline]

Description

Gatsby builds fail when an image (that's not being used) is missing.

 ERROR 

failed to process http://nitflex.lndo.site/sites/default/files/horror-2_0.jpg
HTTPError: Response code 404 (Not Found)

not finished source and transform nodes - 6.125s
failed Remote file download - 0.185s
not finished Downloading remote files - 0.189s

Steps to reproduce

Follow steps in this training material: https://app.gitbook.com/@mariohernandez/s/gatsby/

Expected result

Builds continue even if an image is missing and no longer in use.

Actual result

Builds fail when images are unused and deleted.

Environment

  System:
    OS: macOS 10.15.7
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 12.13.0 - ~/.nvm/versions/node/v12.13.0/bin/node
    npm: 6.12.0 - ~/.nvm/versions/node/v12.13.0/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 89.0.4389.128
    Firefox: 81.0.2
    Safari: 14.0.3
  npmPackages:
    gatsby: ^2.32.3 => 2.32.11 
    gatsby-image: ^2.2.29 => 2.11.0 
    gatsby-plugin-react-helmet: ^3.1.13 => 3.10.0 
    gatsby-plugin-sass: ^2.1.20 => 2.8.0 
    gatsby-plugin-sharp: ^2.2.31 => 2.14.3 
    gatsby-source-drupal: ^3.2.38 => 3.13.1 
    gatsby-source-filesystem: ^2.1.41 => 2.11.1 
    gatsby-transformer-sharp: ^2.2.23 => 2.12.1 
  npmGlobalPackages:
    gatsby-cli: 3.3.0

[LekoArts]

Hi!

I'm sorry but I can't follow the reproduction steps as to how you get the error you're seeing. I'm also not sure if this is actually a Gatsby bug as images are only downloaded when they exist so maybe you request it somewhere else?

The guide you're also following is a bit outdated (package version-wise) so you might also try it with Gatsby v3 and latest versions. Maybe reach out to the authors here? https://github.com/mariohernandez/gatsby_training

[mcmadeline]

Thanks for taking a look. I'm also experiencing this issue on another project that is up to date with Gatsby 3 + all plugins. I can update this repo for a better example.

I attended this training with the creator and he experienced this issue during the training. I thought this would be a good repo to open the issue with since it is a lot smaller than our projects.

[thornyweb]

Hello,

I'm trying to upgrade to v3 from v2 and seeing similar behaviour, also using Drupal.

Current working build, locally and in prod on v2 using the following packages:

"gatsby": "^2.13.22",
"gatsby-cli": "^2.7.14",
"gatsby-image": "^2.2.6",
"gatsby-plugin-manifest": "^2.2.3",
"gatsby-plugin-react-helmet": "^3.1.2",
"gatsby-plugin-remove-trailing-slashes": "^2.6.0",
"gatsby-plugin-robots-txt": "^1.5.0",
"gatsby-plugin-sentry": "^1.0.1",
"gatsby-plugin-sharp": "^2.2.7",
"gatsby-plugin-sitemap": "^2.2.22",
"gatsby-plugin-styled-components": "^3.1.2",
"gatsby-source-drupal": "^3.2.7",
"gatsby-source-filesystem": "^2.1.5",
"gatsby-transformer-sharp": "^2.2.3",

But breaks when I upgrade to v3 packages

"gatsby": "^3.3.0",
"gatsby-cli": "^3.3.0",
"gatsby-image": "^3.3.0",
"gatsby-plugin-manifest": "^3.3.0",
"gatsby-plugin-react-helmet": "^4.3.0",
"gatsby-plugin-remove-trailing-slashes": "^3.3.0",
"gatsby-plugin-robots-txt": "^1.5.6",
"gatsby-plugin-sentry": "^1.0.1",
"gatsby-plugin-sharp": "^3.3.0",
"gatsby-plugin-sitemap": "^3.3.0",
"gatsby-plugin-styled-components": "^4.3.0",
"gatsby-source-drupal": "^4.3.0",
"gatsby-source-filesystem": "^3.3.0",
"gatsby-transformer-sharp": "^3.3.0",

The following happens in console on the v3 packages when I run gatsby develop, as soon as a file that no longer exists is referenced the gatsby-source-drupal plugin is throwing and dying which doesn't happen in my current live version.

success createSchemaCustomization - 0.005s
info Starting to fetch all data from Drupal
success Fetch all data from Drupal - 59.256s
info Downloading remote files from Drupal
 ERROR 
(node:54800) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
(Use `node --trace-deprecation ...` to show where the warning was created)
 ERROR 
failed to process https://cdn.../file.png
HTTPError: Response code 404 (Not Found)
failed Remote file download - 0.289s
warn The gatsby-source-drupal plugin has generated no Gatsby nodes. Do you need it?

[LekoArts]

Okay, thanks both for the message!

I assumed this was a local issue so I moved it. If you can provide a minimal reproduction please open a new issue in our issue tracker with a link to that. The training you linked and the other output of packages is a bit more than a minimal reproduction.

This might be an issue with the Drupal source plugin, but to be sure we need to eliminate any other package that might cause this. Thanks!

[thornyweb]

Thanks @LekoArts, I'm trying to get a minimal repro working as easily as I can, as per your instruction, when I have it I'll open an issue. I did manage to reproduce it on a very basic starter in codepen so just need to get that shared.

@mcmadeline I've found a workaround in the mean time which may or may not be helpful to you.

In the gatsby-source-drupal config in gatsby-config.js I excluded all file--file content types using the disallowedLinkTypes. Then for each image field I added it in the filters as an include which means that only the files used in the published pages will be downloaded.

e.g.

plugins: [
    {
      // Drupal, don't download all files by default, explicitly only get all images
      resolve: "gatsby-source-drupal",
      options: {
        baseUrl: "***",
        concurrentFileRequests: 50,
        disallowedLinkTypes: [
          `self`,
          `describedby`,
          `file--file`
        ],
        filters: {
          "node--author": "include=field_image",
          "node--template_6_404": "include=field_main_404_image",
          "paragraph--50_50_section": "include=field_main_image",
          "paragraph--about_us_products": "include=field_image_1,field_mobile_image_1",
          "paragraph--app_image": "include=field_image",
        }
      },
    },
]

It's a bit cumbersome and will need managing every time an image field is added or changed in any of the content types in Drupal, but hopefully that wont be too frequent.

Once I had in the exclude for file--file the CLI for Gatsby spat out all the places which were erroring which gave me the complete list of filters I needed to add (nearly 50 in my case!)

[mcmadeline]

Thank you so much for this @thornyweb! I appreciate you sharing your solution. I'll give it a try.