Remove got dependency from gatsby-source-filesystem as it doesn't properly handle proxy

[TheGuit]

Summary

gatsby-source-filesystem use a dependency to download file, got, this module doesn't work behind http/https proxy. Gatsby use axios elsewhere and you have juste to define two env variables http_proxy and https_proxy to work with proxy.

Relevant information

The author of got refuse to respect this convention, and don't want to implement proxy mechanism. Today the only solution is to use a dirty global agent. Node.js teams already explain why the proxy part wouldn't be handled in core.

What is the main reason of using got (Axios can handle files, or good old request) ?

One dependency off and it would be easier for everyone using a proxy.

[sidharthachatterjee]

That sounds like a reaonsable improvement. Happy to accept a PR!

[thehanimo]

Did a quick search and there are 5 files that use got:

• gatsby-plugin-sharp/src/scheduler.js
• gatsby-plugin-sharp/src/__tests__/scheduler.js
• gatsby-source-filesystem/src/create-remote-file-node.js
• gatsby-source-filesystem/src/__tests__/create-remote-file-node.js
• gatsby-plugin-checker/index.js

Wondering if axios is a drop-in replacement for got though?

[wardpeet]

Let's discuss this a bit more before accepting PRs. I'm a fan of Got, it's well maintained and it does have proxy support.

[TheGuit]

Being fan of got is not an argument.

What is the pros of got versus axios in this case ? Is those pros really needed ?

The use made of got in Gatsby involve to make use of global-agent to work with proxies, which is dirty (I have to clean my eyes with bleach).

What did you propose to fix this ? The author of got seems to not be open to support proxy like everyone else, so patching got seems not an option.

[LekoArts]

https://github.com/sindresorhus/got#comparison

We're using the Stream API and might want to use the Retries in the future, too. Can you please link the issues with the proxy support?

[TheGuit]

sindresorhus/got#560
sindresorhus/got#79 where it mention nodejs/node#8381

The axios library seems to handle stream now, or i'm not reading it the right way ?

// GET request for remote image in node.js
axios({
  method: 'get',
  url: 'http://bit.ly/2mTM3nY',
  responseType: 'stream'
})
  .then(function (response) {
    response.data.pipe(fs.createWriteStream('ada_lovelace.jpg'))
  });

from the README of Axios

Or other ressources :

• https://github.com/andrewstart/axios-streaming/blob/master/axios.js
• https://futurestud.io/tutorials/download-files-images-with-axios-in-node-js#implementtheaxiosfiledownloadinnodejs

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[ariadne-github]

Not stale! I think got can use a proxy, only that the wrapper module gatsby-source-filesystem is not configured to use it. So even gatsby-source-contentful for example, that depends on gatsby-source-filesystem to download the images for placeholders, does not work behind a proxy.