gatsby-source-filesystem: create-remote-file-node does not handle retries

[schoeneu]

Description

In https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L162 , gatsby-source-filesystem tries to set up retries in case the remote connection breaks when creating a new file node.

However, the "got" documentation states that for "got.stream", the retry-option is ignored:
https://github.com/sindresorhus/got/blob/master/readme.md#retry

Note: When using streams, this option is ignored. If the connection is reset when downloading, you need to catch the error and clear the file you were writing into to prevent duplicated content.

Steps to reproduce

Use a plugin that uses create-remote-file-node (for example remote-images) with a flaky server giving 502s sometimes. No retries will happen.

Expected result

5xx errors from server lead to retries.

Actual result

Instant breakage.

[vladar]

Thank you for opening this, @schoeneu

Gatsby uses got@^8.3.2 branch which doesn't have this quote in docs. But you are probably right and even if version 8 of got supports retries for streams - they are probably broken anyway. Here is a quote from one of the got issues about this (sindresorhus/got#871 (comment)):

This is because the connection may get broken while downloading, and if the retry functionality was here you would get duplicated content. To "retry" properly you need to make your own function that will handle the errors - then make new stream and pipe it again (if you were writing to a file, don't forget to clear it).

The point is that Got cannot clear the file you were writing into.

Also related: sindresorhus/got#309

Which makes sense. So we do need to account for this in gatsby-source-filesystem. PR to address this is more than welcome! 💜

[axe312ger]

This PR would benefit from a fix on this as well: #24288

It would also allow #15783 to use createRemoteFile again :)

[axe312ger]

@pieh I think this ticket was about retry not working. The PR now removed the (broken) retry.

Still, if we get a network error, create-remote-file-node will still break the whole build instead of retrying :(

IMHO this is not fixed

[pieh]

Oh, I didn't notice the PR was linked to the issue, re-opening as functionally nothing changed

[axe312ger]

@pieh ohh 👌

[axe312ger]

@LekoArts I am pretty sure that create-remote-file-node will retry on Gatsby v3!

[wardpeet]

This is fixed in the latest gatsby-source-filesystem@3 version. We added some extra logic to retry when we get partial responses.

[axe312ger]

So we can close this discussion? :)