Update retry algorithm to be more robust

[egrace479]

That being said, retry algorithms (at least robust ones) for internet protocols are normally written with an exponential escalation of wait time (such as 1, 2, 4, 8, 16, 32, .. seconds). In that case, a user may want to specify at which point to give up and log a failure, for example --max-retries and/or --max-wait.

Originally posted by @hlapp in #1 (comment)

reply: It's waiting after a failed attempt if the response is any of the following: 429, 500, 502, 503, 504. It doesn't have a wait between successful downloads. It has a max number of times to retry on the designated responses, but otherwise just logs the response in the error log (along with the index, filename, and url).

Setting a maximum wait time on a request would probably be a good idea as well. urllib3.request seems to handle much of this when also passed a Retry object. @thompsonmj had also suggested HTTPAdapter as an option that also uses Retry.

Seems reasonable to use HTTPAdapter, since it's already using requests. Must also consider streaming interruption, as noted here.

[johnbradley]

The request HTTPAdapter with the urllib3 Retry strategy looks good for some of the retry needs. The streaming interruption will still need to be handled separately though.

[johnbradley]

Sometimes when downloading files we end up reaching a threshold where our IP address gets blocked for a while by a remote server. In that case you typically have to wait for a few hours. I wouldn't expect or want the command to wait in this scenario. For that scenario can we re-run the cautious-robot command and have it skip already downloaded images?

[egrace479]

Sometimes when downloading files we end up reaching a threshold where our IP address gets blocked for a while by a remote server. In that case you typically have to wait for a few hours. I wouldn't expect or want the command to wait in this scenario. For that scenario can we re-run the cautious-robot command and have it skip already downloaded images?

Right now I believe it relies on adjusting the start index to avoid re-downloading the image. However, I could add a line here checking for the image:

if os.path.exists(image_dir_path/image_name):
    continue

[thompsonmj]

From Copilot on #40 regarding the extension equivalency check:

The HEAD request lacks retry logic and could fail on temporary network issues. Consider adding basic retry logic or handling common exceptions to improve reliability when inferring content types.

This is effectively two requests per URL in cases where the extension must be drawn from actual data. Could use retries or another approach relevant to #3.

[thompsonmj]

Additionally from Copilot on #40:

The exponential backoff calculation could result in very long wait times. For example, with default retry=5 and wait=3, the final attempt would wait 48 seconds (3 * 2^4). Consider adding a maximum backoff cap to prevent excessive delays.

[egrace479]

Exponential backoff was partially implemented in #40. It was just for 403 errors (reason and implementation).

See added notes about from @thompsonmj (& Copilot).