Handle temporary web3.storage outages

[bajtos]

From time to time, when we fetch CID from the web3.storage gateway, we encounter an error like this one:

Error: Invalid block bafkreiateg65hcueh66ytp6yufwaihpnvw4oyy27d3krdhj2hv4wdr2ssu of root bafybeigfpytfzw23e2dj6ebomyhmxbd6eqfuorpknqidizriwxldq6x5xm
    at Object.get (file:///home/runner/work/spark-evaluate/spark-evaluate/lib/preprocess.js:150:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///home/runner/work/spark-evaluate/spark-evaluate/node_modules/ipfs-unixfs-exporter/dist/src/resolvers/unixfs-v1/content/file.js:60:27 {

  [cause]: TypeError: Cannot read properties of undefined (reading 'cid')
      at validateBlock (file:///home/runner/work/spark-evaluate/spark-evaluate/node_modules/@web3-storage/car-block-validator/esm/src/index.js:56:36)
      at Object.get (file:///home/runner/work/spark-evaluate/spark-evaluate/lib/preprocess.js:148:15)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async file:///home/runner/work/spark-evaluate/spark-evaluate/node_modules/ipfs-unixfs-exporter/dist/src/resolvers/unixfs-v1/content/file.js:60:27
}

In #134, we implemented a retry scheme to work around the problem.

However, in recent dry-run evaluations in GitHub Actions (see e.g. #196), I noticed that even 10 retries are not enough to fetch the CAR file.

• Is this a problem specific to GHA only, or does it happen in production too?
• What other measures can we implement to avoid failures in this component of our service?

[juliangruber]

I don't see how this could be specific to GHA only 🤔 Can we just increase the retry count?