Persistent image hashes using createRemoteFileNode

[jakubjafra]

Hi,

I was wondering if there is something I can do to not regenerate all images all the time. I have a rather big amount of remote images (right now around 1.5k; it will only grow) that I need to be optimised for my Gatsby site. Currently, each time I run gatsby develop or gatsby build, the images are being regenerated. This can take up 2 minutes right now. Another problem is that each time I run gatsby build new set of images are being generated (with different hashes), so after I deploy the site, the user needs to download a completely different set of images. But he shouldn't - the images stayed the same.

I'm using gatsby-plugin-remote-images with createResolvers workaround for gatsby-source-graphql (I've found it somewhere, and it just works):

// in gatsby-node.js:
module.exports.createResolvers = async ({
  actions,
  cache,
  createNodeId,
  createResolvers,
  store,
  reporter,
}) => {
  const { createNode } = actions;
  await createResolvers({
    SomeType: {
      localPictureUrl: {
        type: 'File',
        async resolve(source) {
          return await createRemoteFileNode({
            url: encodeURI(source.pictureUrl),
            store,
            cache,
            createNode,
            createNodeId,
            reporter,
          });
        },
      },
    },
  });
};

So, the problems I'm trying to find a solution for, are:

1. long build times due to the need to regenerate all images
2. cache issue for the end-users

Do you guys have any ideas on how I could make this work?

Thanks!

[vladar]

Does your web server that serves images supports etag header? AFAIK Gatsby uses them for remote file caching. I mean Gatsby will still send HTTP requests with appropriate If-None-Match header but as long as your image server returns the correct cached status code, i.e. 304 Not Modified, Gatsby shouldn't be re-downloading the image contents.

Also, quick googling shows the following "manual" alternative if your server doesn't support etag: https://thoughtsandstuff.com/gatsby-with-wordpress-caching-downloaded-media-images-to-reduce-build-time/

We do have plans to improve caching for remote files and image transformations in one of the next versions.

In the meantime for development, try using lazy image transformations and query on demand features together. Enable them this way (since [email protected]):

// gatsby-config.js
module.exports = {
  flags: {
    QUERY_ON_DEMAND: true,
    LAZY_IMAGES: true,
  },

And let us know if you see any problems.

[jakubjafra]

Sorry for my late response, but I needed to modify my server to return the ETag header (I'm serving it from S3 via Cloudfront, and something was not working).

Anyway, enabling the ETag header on the origin had 2 effects:

• On next production build gatsby had rebuilt all images - it changed the naming of the generated files (so I needed to reupload them to the hosting provider). I think that was what you meant by "Gatsby uses them for remote file caching"?
• When starting development server, or with every production build I can still see "Downloading remote images..." regenerating every image (based on the number of images handled). It sometimes gets stuck at the end (but only for development), like: 1894/1899 100% Downloading remote files, indefinitely (or just beyond my patience), for no apparent reason whatsoever.