Don't fetch from S3 if cache exists.

[lightyaer]

Description

I have a million json objects in S3 and, on build I have cached them, I don't want to refetch all of them if cache exists, or maybe just fetch the ones which are new/updated in S3.

It induces unnecessary costs on S3 io, since each JSON object is around 50kb to 250kb

PS: I haven't used any plugin to get json objects from S3. Used aws-sdk in gatsby-node.js to fetch the objects.

My plan is to have a box fetch from SCM, build only the transient parts(new/updated jsons), which I get from an S3 event, deploy
to a deployment bucket in S3.

Can anyone let me know if this is possible, I have used gatsby-plugin-tidy, so I have to only add/update items to page-data and [route] folder in public.

Environment

System:
OS: Linux 5.4 Ubuntu 20.04.1 LTS (Focal Fossa)
CPU: (8) x64 Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
Shell: 5.8 - /usr/bin/zsh
Binaries:
Node: 12.18.3 - ~/.nvm/versions/node/v12.18.3/bin/node
Yarn: 1.22.5 - ~/.nvm/versions/node/v12.18.3/bin/yarn
npm: 6.14.6 - ~/.nvm/versions/node/v12.18.3/bin/npm
Languages:
Python: 2.7.18 - /usr/bin/python
Browsers:
Chrome: 87.0.4280.66
Firefox: 83.0
npmPackages:
gatsby: ^2.25.3 => 2.25.3
gatsby-image: ^2.4.21 => 2.4.21
gatsby-plugin-manifest: ^2.5.2 => 2.5.2
gatsby-plugin-material-ui: ^2.1.10 => 2.1.10
gatsby-plugin-offline: ^3.3.2 => 3.3.2
gatsby-plugin-react-helmet: ^3.3.14 => 3.3.14
gatsby-plugin-sharp: ^2.7.1 => 2.7.1
gatsby-plugin-tidy: ^1.0.1 => 1.0.1
gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4
gatsby-source-filesystem: ^2.4.2 => 2.4.2
gatsby-transformer-json: ^2.4.15 => 2.4.15
gatsby-transformer-sharp: ^2.5.21 => 2.5.21
npmGlobalPackages:
gatsby-cli: 2.12.117