Create File image from external images URL on onCreateNode

[lezan]

Hello,
I am try to create a File to use images sourced from external image URLs but I am not able to do it correctly, because running gatsby develop always give me this error:

Error: EPERM: operation not permitted, lstat '.cache\caches\default-site-plugin\tmp-1e3da4727bb3ae064a6cf091adc0ab46.jpg'
at lstatSync (node:fs:1514:3)
at rimrafSync (node:internal/fs/rimraf:180:13)
at Object.rmSync (node:fs:1214:10)
at Object.removeSync (\node_modules\fs-extra\lib\remove\index.js:15:28)
at WriteStream. (\node_modules\gatsby-core-utils\dist\fetch-remote-file.js:122:26)
at WriteStream.emit (node:events:402:35)
at finish (node:internal/streams/writable:750:10)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -4048,
syscall: 'lstat',
code: 'EPERM',
path: '****\.cache\caches\default-site-plugin\tmp-1e3da4727bb3ae064a6cf091adc0ab46.jpg'
}

I am doing it for two different fields, a string and an array of string. I am able to correctly create File node with the single string, but with array of string I have some issues.

exports.onCreateNode = async ({
  node,
  actions: { createNode, createNodeField },
  createNodeId,
  cache,
  store,
}) => {
  // Single string
  if (node.internal.type === 'MyType') {
    const fileNodeImageMap = await createRemoteFileNode({
      url: node.imageLink,
      parentNodeId: node.id,
      createNode,
      createNodeId,
      cache,
      store,
    });

    if (fileNodeImageMap) {
      createNodeField({ node, name: 'imageLinkFile', value: fileNodeImageMap.id });
    }

    // Array of string
    const links = node.additionalImageLink.split(',');
    const additionalImages = links.map(async (d) => {
      const fileNodeImage = await createRemoteFileNode({
        url: d,
        parentNodeId: node.id,
        createNode,
        createNodeId,
        cache,
        store,
      });

      if (fileNodeImage) {
        return fileNodeImage.id;
      }
    });

    if (additionalImages.length > 0) {
      const responses = await Promise.all(additionalImages);
      const additionalId = responses.map((d) => d.id);
      createNodeField({ node, name: 'additionalImageLinkFile', value: additionalId });
    }
  }
};

Can you point me what I am doing wrong? Thanks!

[lezan]

Some news.

I tried with random image, with caching and without caching.
If I use https://source.unsplash.com/random/1920x1080 images are going to be cached and I am able to start dev mode without error.
Without images caching and using https://source.unsplash.com/random/1920x1080${Math.random()} I get the same error as before.
I am pretty sure now it is a timeout issue.
I have only 300 images, is it a problem?

[lezan]

Setting some env variables to GATSBY_CONCURRENT_DOWNLOAD=1,GATSBY_CONNECTION_TIMEOUT=200000 and GATSBY_STALL_TIMEOUT=200000 I am now able to start dev mode for the first time. I think the issue (maybe a bug?) is related to a try to remove an element (a tmp image) not present (not fully download).
It checks if (!haveAllBytesBeenWritten) adn then try to remove temp file with _fsExtra.default.removeSync(tmpFilename); haveAllBytesBeenWritten is false because it not fully written, maybe because I get a connection timeout. Still not sure why tmp does not exists so removing it give error.